Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-6735

A minor optimization to avoid pread() be blocked by read() inside the same DFSInputStream

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 3.0.0-alpha1
    • 2.7.0
    • hdfs-client
    • None
    • Reviewed

    Description

      In current DFSInputStream impl, there're a couple of coarser-grained locks in read/pread path, and it has became a HBase read latency pain point so far. In HDFS-6698, i made a minor patch against the first encourtered lock, around getFileLength, in deed, after reading code and testing, it shows still other locks we could improve.
      In this jira, i'll make a patch against other locks, and a simple test case to show the issue and the improved result.

      This is important for HBase application, since in current HFile read path, we issue all read()/pread() requests in the same DFSInputStream for one HFile. (Multi streams solution is another story i had a plan to do, but probably will take more time than i expected)

      Attachments

        1. HDFS-6735.txt
          9 kB
          Liang Xie
        2. HDFS-6735-v2.txt
          13 kB
          Liang Xie
        3. HDFS-6735-v3.txt
          11 kB
          Lars Hofhansl
        4. HDFS-6735-v4.txt
          22 kB
          Lars Hofhansl
        5. HDFS-6735-v5.txt
          22 kB
          Lars Hofhansl
        6. HDFS-6735-v6.txt
          23 kB
          Lars Hofhansl
        7. HDFS-6735-v7.txt
          22 kB
          Lars Hofhansl
        8. HDFS-6735-v8.txt
          26 kB
          Lars Hofhansl

        Issue Links

          is related to

          Improvement - An improvement or enhancement to an existing feature or task. HADOOP-15557 CryptoInputStream can't handle concurrent access; inconsistent with HDFS

          • Major - Major loss of function.
          • Open
          relates to

          Bug - A problem which impairs or prevents the functions of the product. HDFS-7494 Checking of closed in DFSInputStream#pread() should be protected by synchronization

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. HDFS-7495 Remove updatePosition argument from DFSInputStream#getBlockAt()

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. HADOOP-9361 Strictly define the expected behavior of filesystem APIs and write tests to verify compliance

          • Blocker - Blocks development and/or testing work, production could not run
          • Closed

          Activity

            People

              larsh Lars Hofhansl
              xieliang007 Liang Xie
              Votes:
              0 Vote for this issue
              Watchers:
              17 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: