Details

    • Sub-task
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 3.0.0-alpha1
    • 2.3.0
    • datanode
    • None
    • Reviewed

    Description

      The fix for HDFS-5484 was accidentally regressed by the following change made via HDFS-5542

      +  DatanodeStorageInfo updateStorage(DatanodeStorage s) {
           synchronized (storageMap) {
             DatanodeStorageInfo storage = storageMap.get(s.getStorageID());
             if (storage == null) {
      @@ -670,8 +658,6 @@
                        " for DN " + getXferAddr());
               storage = new DatanodeStorageInfo(this, s);
               storageMap.put(s.getStorageID(), storage);
      -      } else {
      -        storage.setState(s.getState());
             }
             return storage;
           }
      

      By removing the 'else' and no longer updating the state in the BlockReport processing path, we effectively get the bogus state & type that is set via the first heartbeat (see the fix for HDFS-5455):

      +      if (storage == null) {
      +        // This is seen during cluster initialization when the heartbeat
      +        // is received before the initial block reports from each storage.
      +        storage = updateStorage(new DatanodeStorage(report.getStorageID()));
      

      Even reverting the change and reintroducing the 'else' leaves the state & type temporarily inaccurate until the first block report.

      As discussed with arpitagarwal, a better fix would be to simply include the full DatanodeStorage object in the StorageReport (as opposed to only the Storage ID). This requires adding the DatanodeStorage object to StorageReportProto. It needs to be a new optional field and we cannot remove the existing StorageUuid for protocol compatibility.

      Attachments

        1. h5667.05.patch
          31 kB
          Arpit Agarwal
        2. h5667.04.patch
          32 kB
          Arpit Agarwal
        3. h5667.03.patch
          31 kB
          Arpit Agarwal
        4. h5667.02.patch
          31 kB
          Arpit Agarwal

        Issue Links

          Activity

            People

              arp Arpit Agarwal
              sirianni Eric Sirianni
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: