Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-7701

Opening regions on dead server are not reassigned quickly

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 0.95.2
    • 0.95.0
    • None
    • None
    • Reviewed

    Description

      Closed regions are not removed from assignments. I am not sure if it's a general state problem, or just a small bug; for now, one manifestation is that moved region is ignored by SSH of the target server if target server dies before updating ZK.

      2013-01-22 17:59:00,524 DEBUG [IPC Server handler 3 on 50658] master.AssignmentManager(1475): Sent CLOSE to 10.11.2.92,51231,1358906285048 for region IntegrationTestRebalanceAndKillServersTargeted,66666660,1358906196709.0200b366bc37c5afd1185f7d487c7dfb.
      2013-01-22 17:59:00,997 DEBUG [RS_CLOSE_REGION-10.11.2.92,51231,1358906285048-1] handler.CloseRegionHandler(167): set region closed state in zk successfully for region IntegrationTestRebalanceAndKillServersTargeted,66666660,1358906196709.0200b366bc37c5afd1185f7d487c7dfb. sn name: 10.11.2.92,51231,1358906285048
      2013-01-22 17:59:01,088 INFO  [MASTER_CLOSE_REGION-10.11.2.92,50658,1358906192673-0] master.RegionStates(242): Region {NAME => 'IntegrationTestRebalanceAndKillServersTargeted,66666660,1358906196709.0200b366bc37c5afd1185f7d487c7dfb.', STARTKEY => '66666660', ENDKEY => '7333332c', ENCODED => 0200b366bc37c5afd1185f7d487c7dfb,} transitioned from {IntegrationTestRebalanceAndKillServersTargeted,66666660,1358906196709.0200b366bc37c5afd1185f7d487c7dfb. state=CLOSED, ts=1358906341087, server=null} to {IntegrationTestRebalanceAndKillServersTargeted,66666660,1358906196709.0200b366bc37c5afd1185f7d487c7dfb. state=OFFLINE, ts=1358906341088, server=null}
      2013-01-22 17:59:01,128 INFO  [MASTER_CLOSE_REGION-10.11.2.92,50658,1358906192673-0] master.AssignmentManager(1596): Assigning region IntegrationTestRebalanceAndKillServersTargeted,66666660,1358906196709.0200b366bc37c5afd1185f7d487c7dfb. to 10.11.2.92,50661,1358906192942
      
      ... (50661 didn't update ZK to OPEN, only OPENING)
      
      2013-01-22 17:59:06,605 INFO  [MASTER_SERVER_OPERATIONS-10.11.2.92,50658,1358906192673-2] handler.ServerShutdownHandler(202): Reassigning 7 region(s) that 10.11.2.92,50661,1358906192942 was carrying (skipping 0 regions(s) that are already in transition)
      2013-01-22 17:59:06,605 DEBUG [MASTER_SERVER_OPERATIONS-10.11.2.92,50658,1358906192673-2] handler.ServerShutdownHandler(219): Skip assigning region IntegrationTestRebalanceAndKillServersTargeted,66666660,1358906196709.0200b366bc37c5afd1185f7d487c7dfb. because it has been opened in 10.11.2.92,51231,1358906285048
      

      Note the server in the last line - the one that has long closed the region.

      Attachments

        1. TEST-org.apache.hadoop.hbase.IntegrationTestRebalanceAndKillServersTargeted.xml
          7.38 MB
          Sergey Shelukhin
        2. trunk-7701_v1.patch
          8 kB
          Jimmy Xiang
        3. trunk-7701_v2.patch
          10 kB
          Jimmy Xiang

        Issue Links

          Activity

            People

              jxiang Jimmy Xiang
              sershe Sergey Shelukhin
              Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: