Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-2465

Add HDFS support for fadvise readahead and drop-behind

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 0.23.0
    • 0.23.0, 1.1.0
    • datanode, performance
    • None
    • Reviewed
    • Hide
      HDFS now has the ability to use posix_fadvise and sync_data_range syscalls to manage the OS buffer cache. This support is currently considered experimental, and may be enabled by configuring the following keys:
      dfs.datanode.drop.cache.behind.writes - set to true to drop data out of the buffer cache after writing
      dfs.datanode.drop.cache.behind.reads - set to true to drop data out of the buffer cache when performing sequential reads
      dfs.datanode.sync.behind.writes - set to true to trigger dirty page writeback immediately after writing data
      dfs.datanode.readahead.bytes - set to a non-zero value to trigger readahead for sequential reads
      Show
      HDFS now has the ability to use posix_fadvise and sync_data_range syscalls to manage the OS buffer cache. This support is currently considered experimental, and may be enabled by configuring the following keys: dfs.datanode.drop.cache.behind.writes - set to true to drop data out of the buffer cache after writing dfs.datanode.drop.cache.behind.reads - set to true to drop data out of the buffer cache when performing sequential reads dfs.datanode.sync.behind.writes - set to true to trigger dirty page writeback immediately after writing data dfs.datanode.readahead.bytes - set to a non-zero value to trigger readahead for sequential reads

    Description

      This is the HDFS side of HADOOP-7714. The initial implementation is heuristic based and should be considered experimental, as discussed in the parent JIRA. It should be off by default until better heuristics, APIs, and tuning experience is developed.

      Attachments

        1. HDFS-2465.branch-1.patch
          16 kB
          Brandon Li
        2. hdfs-2465.txt
          16 kB
          Todd Lipcon
        3. hdfs-2465.txt
          16 kB
          Todd Lipcon
        4. hdfs-2465.txt
          16 kB
          Todd Lipcon
        5. hdfs-2465.txt
          16 kB
          Todd Lipcon

        Issue Links

          is blocked by

          Sub-task - The sub-task of the issue HADOOP-7753 Support fadvise and sync_data_range in NativeIO, add ReadaheadPool class

          • Major - Major loss of function.
          • Closed
          is part of

          Bug - A problem which impairs or prevents the functions of the product. HADOOP-7714 Umbrella for usage of native calls to manage OS cache and readahead

          • Major - Major loss of function.
          • Open
          is related to

          Bug - A problem which impairs or prevents the functions of the product. HDFS-2751 Datanode drops OS cache behind reads even for short reads

          • Major - Major loss of function.
          • Closed

          New Feature - A new feature of the product, which has yet to be developed. HDFS-4184 Add the ability for Client to provide more hint information for DataNode to manage the OS buffer cache more accurate

          • Major - Major loss of function.
          • Resolved
          relates to

          Improvement - An improvement or enhancement to an existing feature or task. HDFS-3228 Use fadvise in local read path

          • Major - Major loss of function.
          • Open

          Activity

            People

              tlipcon Todd Lipcon
              tlipcon Todd Lipcon
              Votes:
              0 Vote for this issue
              Watchers:
              20 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: