Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-4350

Make enabling of stale marking on read and write paths independent

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 3.0.0-alpha1
    • 1.2.0, 2.0.3-alpha
    • None
    • None
    • Incompatible change, Reviewed
    • Hide
      This patch makes an incompatible configuration change, as described below:
      In releases 1.1.0 and other point releases 1.1.x, the configuration parameter "dfs.namenode.check.stale.datanode" could be used to turn on checking for the stale nodes. This configuration is no longer supported in release 1.2.0 onwards and is renamed as "dfs.namenode.avoid.read.stale.datanode".

      How feature works and configuring this feature:
      As described in HDFS-3703 release notes, datanode stale period can be configured using parameter "dfs.namenode.stale.datanode.interval" in seconds (default value is 30 seconds). NameNode can be configured to use this staleness information for reads using configuration "dfs.namenode.avoid.read.stale.datanode". When this parameter is set to true, namenode picks a stale datanode as the last target to read from when returning block locations for reads. Using staleness information for writes is as described in the releases notes of HDFS-3912.
      Show
      This patch makes an incompatible configuration change, as described below: In releases 1.1.0 and other point releases 1.1.x, the configuration parameter "dfs.namenode.check.stale.datanode" could be used to turn on checking for the stale nodes. This configuration is no longer supported in release 1.2.0 onwards and is renamed as "dfs.namenode.avoid.read.stale.datanode". How feature works and configuring this feature: As described in HDFS-3703 release notes, datanode stale period can be configured using parameter "dfs.namenode.stale.datanode.interval" in seconds (default value is 30 seconds). NameNode can be configured to use this staleness information for reads using configuration "dfs.namenode.avoid.read.stale.datanode". When this parameter is set to true, namenode picks a stale datanode as the last target to read from when returning block locations for reads. Using staleness information for writes is as described in the releases notes of HDFS-3912 .

    Description

      Marking of datanodes as stale for the read and write path was introduced in HDFS-3703 and HDFS-3912 respectively. This is enabled using two new keys, DFS_NAMENODE_CHECK_STALE_DATANODE_KEY and DFS_NAMENODE_AVOID_STALE_DATANODE_FOR_WRITE_KEY. However, there currently exists a dependency, since you cannot enable write marking without also enabling read marking, since the first key enables both checking of staleness and read marking.

      I propose renaming the first key to DFS_NAMENODE_AVOID_STALE_DATANODE_FOR_READ_KEY, and make checking enabled if either of the keys are set. This will allow read and write marking to be enabled independently.

      Attachments

        1. hdfs-4350-branch-1-3.patch
          22 kB
          Andrew Wang
        2. hdfs-4350-7.patch
          27 kB
          Andrew Wang
        3. hdfs-4350-branch-1-2.patch
          22 kB
          Andrew Wang
        4. hdfs-4350-6.patch
          27 kB
          Andrew Wang
        5. hdfs-4350-branch-1-1.patch
          22 kB
          Andrew Wang
        6. hdfs-4350-5.patch
          26 kB
          Andrew Wang
        7. hdfs-4350.txt
          26 kB
          Todd Lipcon
        8. hdfs-4350-4.patch
          23 kB
          Andrew Wang
        9. hdfs-4350-3.patch
          22 kB
          Andrew Wang
        10. hdfs-4350-2.patch
          22 kB
          Andrew Wang
        11. hdfs-4350-1.patch
          22 kB
          Andrew Wang

        Issue Links

          relates to

          Sub-task - The sub-task of the issue HDFS-3912 Detecting and avoiding stale datanodes for writing

          • Major - Major loss of function.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. HDFS-3703 Decrease the datanode failure detection time

          • Major - Major loss of function.
          • Closed

          Activity

            People

              andrew.wang Andrew Wang
              andrew.wang Andrew Wang
              Votes:
              0 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: