Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-3528

Use native CRC32 in DFS write path

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 2.0.0-alpha
    • 2.6.0
    • datanode, hdfs-client, performance
    • None

    Description

      HDFS-2080 improved the CPU efficiency of the read path by using native SSE-enabled code for CRC verification. Benchmarks of the write path show that it's often CPU bound by checksums as well, so we should make the same improvement there.

      Attachments

        Issue Links

          breaks

          Bug - A problem which impairs or prevents the functions of the product. HADOOP-11064 UnsatisifedLinkError with hadoop 2.4 JARs on hadoop-2.6 due to NativeCRC32 method changes

          • Blocker - Blocks development and/or testing work, production could not run
          • Closed
          is part of

          Task - A task that needs to be done. HDFS-4225 Improve HDFS write performance

          • Major - Major loss of function.
          • Open
          is related to

          Improvement - An improvement or enhancement to an existing feature or task. HDFS-2080 Speed up DFS read path by lessening checksum overhead

          • Major - Major loss of function.
          • Closed
          relates to

          Improvement - An improvement or enhancement to an existing feature or task. HDFS-5395 write(ByteBuffer) method

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. MAPREDUCE-5962 Support CRC32C in IFile

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. HDFS-3529 Use direct buffers for data in write path

          • Major - Major loss of function.
          • Patch Available
          There are no Sub-Tasks for this issue.

          Activity

            People

              james.thomas James Thomas
              tlipcon Todd Lipcon
              Votes:
              0 Vote for this issue
              Watchers:
              23 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: