Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-17484

Introduce redundancy.considerLoad.minLoad to avoiding excluding nodes when they are not busy actually

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Open
    • Minor
    • Resolution: Unresolved
    • 3.4.0
    • None
    • namenode

    Description

      Currently, we have `dfs.namenode.redundancy.considerLoad` equals true by default, and 

      dfs.namenode.redundancy.considerLoad.factor equals 2.0 by default.

      Think about below situation. when we are doing stress test, we may deploy hdfs client onto the datanode. So, this hdfs client will prefer to write to its local datanode and increase this machine's load.  Suppose we have 3 datanodes, the load of them are as below:  5.0, 0.2, 0.3.

       

      The load equals to 5.0 will be excluded when choose datanodes for a block. But actually, it is not slow node when load equals to 5.0 for a machine with 80 cpu cores.

       

      So, we should better add a new configuration entry :  `dfs.namenode.redundancy.considerLoad.minLoad` to indicate the mininum factor we will make considerLoad take effect.

      Attachments

        Issue Links

          Activity

            People

              zhanghaobo farmmamba
              zhanghaobo farmmamba
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated: