Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-1700

Append to files in HDFS

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 0.15.1
    • 0.19.0
    • None
    • None
    • Incompatible change, Reviewed
    • Hide
      Introduced append operation for HDFS files.
      Show
      Introduced append operation for HDFS files.

    Description

      Request for being able to append to files in HDFS has been raised a couple of times on the list of late. For one example, see http://www.nabble.com/HDFS%2C-appending-writes-status-tf3848237.html#a10916193. Other mail describes folks' workarounds because this feature is lacking: e.g. http://www.nabble.com/Loading-data-into-HDFS-tf4200003.html#a12039480 (Later on this thread, Jim Kellerman re-raises the HBase need of this feature). HADOOP-337 'DFS files should be appendable' makes mention of file append but it was opened early in the life of HDFS when the focus was more on implementing the basics rather than adding new features. Interest fizzled. Because HADOOP-337 is also a bit of a grab-bag – it includes truncation and being able to concurrently read/write – rather than try and breathe new life into HADOOP-337, instead, here is a new issue focused on file append. Ultimately, being able to do as the google GFS paper describes – having multiple concurrent clients making 'Atomic Record Append' to a single file would be sweet but at least for a first cut at this feature, IMO, a single client appending to a single HDFS file letting the application manage the access would be sufficent.

      Attachments

        1. appendtrunk16.patch
          80 kB
          Dhruba Borthakur
        2. appendtrunk15.patch
          80 kB
          Dhruba Borthakur
        3. appendtrunk14.patch
          79 kB
          Dhruba Borthakur
        4. appendtrunk14.patch
          80 kB
          Dhruba Borthakur
        5. appendtrunk13.patch
          79 kB
          Dhruba Borthakur
        6. appendtrunk13.patch
          79 kB
          Dhruba Borthakur
        7. appendtrunk13.patch
          64 kB
          Dhruba Borthakur
        8. appendtrunk12.patch
          80 kB
          Tsz-wo Sze
        9. appendtrunk11.patch
          80 kB
          Tsz-wo Sze
        10. appendtrunk10.patch
          81 kB
          Tsz-wo Sze
        11. appendtrunk9.patch
          79 kB
          Tsz-wo Sze
        12. appendtrunk8.patch
          76 kB
          Tsz-wo Sze
        13. appendtrunk7.patch
          73 kB
          Dhruba Borthakur
        14. appendtrunk6.patch
          73 kB
          Dhruba Borthakur
        15. append3.patch
          55 kB
          Dhruba Borthakur
        16. 1700_20080606.patch
          14 kB
          Tsz-wo Sze
        17. Grid_HadoopRenumberBlocks.pdf
          74 kB
          Robert Chansler
        18. append.patch
          22 kB
          Ruyue Ma
        19. Appends.doc
          76 kB
          Dhruba Borthakur
        20. Appends.html
          45 kB
          Dhruba Borthakur
        21. Appends.doc
          70 kB
          Dhruba Borthakur

        Issue Links

          depends upon

          New Feature - A new feature of the product, which has yet to be developed. HADOOP-3283 Need a mechanism for data nodes to update generation stamps.

          • Major - Major loss of function.
          • Closed

          New Feature - A new feature of the product, which has yet to be developed. HADOOP-3310 Lease recovery for append

          • Major - Major loss of function.
          • Closed
          is blocked by

          Bug - A problem which impairs or prevents the functions of the product. HADOOP-2565 DFSPath cache of FileStatus can become stale

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HADOOP-2655 Copy on write for data and metadata files in the presence of snapshots

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. HADOOP-2656 Support for upgrading existing cluster to facilitate appends to HDFS files

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. HADOOP-3113 DFSOututStream.flush() should flush data to real block file on DataNode.

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. HADOOP-3176 Change lease record when a open-for-write-file gets renamed

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. HADOOP-3503 Race condition when client and namenode start block recovery simultaneously

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. HADOOP-3161 TestFileAppend fails on Mac since HADOOP-2655 was committed

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Closed

          New Feature - A new feature of the product, which has yet to be developed. HADOOP-2658 Design and Implement a Test Plan to support appends to HDFS files

          • Blocker - Blocks development and/or testing work, production could not run
          • Closed

          New Feature - A new feature of the product, which has yet to be developed. HADOOP-3201 namenode should be able to retrieve block metadata from a datanode

          • Major - Major loss of function.
          • Closed

          New Feature - A new feature of the product, which has yet to be developed. HADOOP-3250 Extend FileSystem API to allow appending to files

          • Major - Major loss of function.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. HADOOP-1707 Remove the DFS Client disk-based cache

          • Major - Major loss of function.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. HADOOP-2345 new transactions to support HDFS Appends

          • Major - Major loss of function.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. HADOOP-3177 Expose DFSOutputStream.fsync API though the FileSystem interface

          • Major - Major loss of function.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. HADOOP-3515 Protocol changes to allow appending to the last partial crc chunk of a file

          • Major - Major loss of function.
          • Closed
          is depended upon by

          Test - A new unit, integration or system test. HADOOP-3790 Add more unit tests to test appending to files in HDFS

          • Blocker - Blocks development and/or testing work, production could not run
          • Closed
          is related to

          New Feature - A new feature of the product, which has yet to be developed. HDFS-200 In HDFS, sync() not yet guarantees data available to the new readers

          • Blocker - Blocks development and/or testing work, production could not run
          • Closed

          New Feature - A new feature of the product, which has yet to be developed. HADOOP-337 DFS files should be appendable

          • Major - Major loss of function.
          • Closed
          is superceded by

          Improvement - An improvement or enhancement to an existing feature or task. HDFS-265 Revisit append

          • Major - Major loss of function.
          • Closed
          relates to

          Bug - A problem which impairs or prevents the functions of the product. HADOOP-3834 Checkin the design document for HDFS appends into source control repository

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HADOOP-89 files are not visible until they are closed

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. HADOOP-1497 Possibility of duplicate blockids if dead-datanodes come back up after corresponding files were deleted

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. HADOOP-3241 DFSFileInfo should also have field to say if the file is underconstrction

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. HADOOP-3329 DatanodeDescriptor objects stored in FSImage may be out dated.

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. HADOOP-3832 Create more unit tests for testing HDFS appends

          • Major - Major loss of function.
          • Closed

          New Feature - A new feature of the product, which has yet to be developed. HADOOP-2657 Enhancements to DFSClient to support flushing data at any point in time

          • Major - Major loss of function.
          • Closed

          Test - A new unit, integration or system test. HDFS-2253 Create a benchmark to measure performance of "append" to HDFS files

          • Major - Major loss of function.
          • Resolved

          Activity

            People

              dhruba Dhruba Borthakur
              stack Michael Stack
              Votes:
              11 Vote for this issue
              Watchers:
              28 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: