Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-380

support for persistent connections to improve random read performance.

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Duplicate
    • None
    • None
    • None
    • None

    • Linux 2.6.9-55 , Dual Core Opteron 280 2.4Ghz , 4GB memory

    Description

      preads() establish new connections per request. yourkit java profiles show that this connection overhead is pretty significant on the DataNode.
      I wrote a simple microbenchmark program which does many iterations of pread() from different offsets of a large file. I hacked DFSClient/DataNode code to re-use the same connection/DataNode request handler thread. The performance improvement was 7% when the data is served from disk and 80% when the data is served from the OS page cache.

      Attachments

        1. pread_test.java
          1 kB
          George Wu

        Issue Links

          is related to

          Improvement - An improvement or enhancement to an existing feature or task. HDFS-1599 Umbrella Jira for Improving HBASE support in HDFS

          • Major - Major loss of function.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. HDFS-941 Datanode xceiver protocol should allow reuse of a connection

          • Major - Major loss of function.
          • Closed
          relates to

          Task - A task that needs to be done. HDFS-656 Clarify error handling and retry semantics for DFS read path

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Open

          Activity

            People

              Unassigned Unassigned
              georgewu George Wu
              Votes:
              0 Vote for this issue
              Watchers:
              16 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: