XMLWordPrintableJSON

Details

    Description

      Today, initialization of replication queues blocks safe mode exit and certain HA state transitions. For a big name space, this can take hundreds of seconds with the FSNamesystem write lock held. During this time, important requests (e.g. initial block reports, heartbeat, etc) are blocked.

      The effect of delaying the initialization would be not starting replication right away, but I think the benefit outweighs. If we make it asynchronous, the work per iteration should be limited, so that the lock duration is capped.

      If full/incremental block reports and any other requests that modifies block state properly performs replication checks while the blocks are scanned and the queues populated in background, every block will be processed. (Some may be done twice) The replication monitor should run even before all blocks are processed.

      This will allow namenode to exit safe mode and start serving immediately even with a big name space. It will also reduce the HA failover latency.

      Attachments

        1. HDFS-5496.patch
          15 kB
          Vinayakumar B
        2. HDFS-5496.patch
          14 kB
          Vinayakumar B
        3. HDFS-5496.patch
          18 kB
          Vinayakumar B
        4. HDFS-5496.patch
          19 kB
          Vinayakumar B
        5. HDFS-5496.patch
          19 kB
          Vinayakumar B

        Activity

          People

            vinayakumarb Vinayakumar B
            kihwal Kihwal Lee
            Votes:
            0 Vote for this issue
            Watchers:
            14 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: