Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-16902

Add Namenode status to BPServiceActor metrics and improve logging in offerservice

    XMLWordPrintableJSON

Details

    • Reviewed

    Description

      Recently came across an k8s environment where randomly some datanode pods are not able to stay connected to all namenode pods (e.g. last heartbeat time stays higher than 2 hr sometimes). When any standby namenode becomes active, any datanode that is not heartbeating to it for quite sometime would not be able to send any further block reports, leading to missing replicas immediately after namenode failover, which could only be resolved with datanode pod restart.

      While the issue seems env specific, BPServiceActor's offer service could use some logging improvements. It is also good to get namenode status exposed with BPServiceActorInfo to identify any lags from datanode side in recognizing updated Active namenode status with heartbeats.

      Attachments

        Issue Links

          Activity

            People

              vjasani Viraj Jasani
              vjasani Viraj Jasani
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: