Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-265

Revisit append

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 0.21.0
    • 0.21.0
    • None
    • None

    Description

      HADOOP-1700 and related issues have put a lot of efforts to provide the first implementation of append. However, append is such a complex feature. It turns out that there are issues that were initially seemed trivial but needs a careful design. This jira revisits append, aiming for a design and implementation supporting a semantics that are acceptable to its users.

      Attachments

        1. appendDesign3.pdf
          869 kB
          Hairong Kuang
        2. AppendTestPlan.html
          64 kB
          Konstantin I Boudnik
        3. AppendTestPlan.html
          65 kB
          Konstantin I Boudnik
        4. a.sh
          1 kB
          Tsz-wo Sze
        5. AppendTestPlan.html
          65 kB
          Konstantin I Boudnik
        6. AppendTestPlan.html
          65 kB
          Konstantin I Boudnik
        7. AppendTestPlan.html
          65 kB
          Konstantin I Boudnik
        8. AppendTestPlan.html
          63 kB
          Konstantin I Boudnik
        9. appendDesign2.pdf
          868 kB
          Hairong Kuang
        10. AppendTestPlan.html
          62 kB
          Konstantin I Boudnik
        11. AppendTestPlan.html
          62 kB
          Konstantin I Boudnik
        12. AppendTestPlan.html
          55 kB
          Konstantin I Boudnik
        13. appendDesign1.pdf
          624 kB
          Hairong Kuang
        14. TestPlanAppend.html
          50 kB
          Konstantin I Boudnik
        15. appendDesign.pdf
          408 kB
          Konstantin Shvachko
        16. appendDesign.pdf
          580 kB
          Hairong Kuang
        17. AppendSpec.pdf
          48 kB
          Hairong Kuang

        Issue Links

          incorporates

          Improvement - An improvement or enhancement to an existing feature or task. HDFS-728 Create a comprehensive functional test for append

          • Major - Major loss of function.
          • Closed
          is related to

          New Feature - A new feature of the product, which has yet to be developed. HADOOP-6239 Command-line for append

          • Major - Major loss of function.
          • Resolved

          New Feature - A new feature of the product, which has yet to be developed. HDFS-744 Support hsync in HDFS

          • Major - Major loss of function.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. HDFS-1831 HDFS equivalent of HADOOP-7223 changes to handle FileContext createFlag combinations

          • Major - Major loss of function.
          • Closed
          relates to

          Bug - A problem which impairs or prevents the functions of the product. HDFS-588 Fix TestFiDataTransferProtocol and TestAppend2 failures in append branch.

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HDFS-1060 Append/flush should support concurrent "tailer" use case

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HDFS-195 Need to handle access token expiration when re-establishing the pipeline for dfs write

          • Major - Major loss of function.
          • Closed

          New Feature - A new feature of the product, which has yet to be developed. HDFS-1606 Provide a stronger data guarantee in the write pipeline

          • Major - Major loss of function.
          • Resolved

          New Feature - A new feature of the product, which has yet to be developed. HADOOP-4359 Access Token: Support for data access authorization checking on DataNodes

          • Major - Major loss of function.
          • Closed

          Test - A new unit, integration or system test. HDFS-714 Create fault injection test for the new pipeline close

          • Major - Major loss of function.
          • Closed

          Test - A new unit, integration or system test. HDFS-451 Test DataTransferProtocol with fault injection

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Closed
          supercedes

          New Feature - A new feature of the product, which has yet to be developed. HADOOP-1700 Append to files in HDFS

          • Major - Major loss of function.
          • Closed
          1.
          		 Factor out BlockInfo from BlocksMap Sub-task Closed Konstantin Shvachko
          2.
          		 Redesign DataNode volumeMap to include all types of Replicas Sub-task Resolved Hairong Kuang
          3.
          		 Introduce BlockInfoUnderConstruction to reflect block replica states while writing. Sub-task Resolved Konstantin Shvachko
          4.
          		 Create new tests for Append's hflush Sub-task Resolved Konstantin I Boudnik
          5.
          		 Create new tests for lease recovery Sub-task Closed Konstantin I Boudnik
          6.
          		 Create new tests for block recovery Sub-task Closed Hairong Kuang
          7.
          		 Create new tests for pipeline Sub-task Closed Konstantin I Boudnik
          8.
          		 Create stress tests for append feature Sub-task Resolved Unassigned
          9.
          		 Support hflush at DFSClient Sub-task Resolved Hairong Kuang
          10.
          		 DataNode uses ReplicaBeingWritten to support dfs writes/hflush Sub-task Resolved Hairong Kuang
          11.
          		 Break FSDatasetInterface#writeToBlock() into writeToTemporary, writeToRBW, and append Sub-task Resolved Hairong Kuang
          12.
          		 Add a "rbw" sub directory to DataNode data directory Sub-task Resolved Hairong Kuang
          13.
          		 DataNode restarts may introduce corrupt/duplicated/lost replicas when handling detached replicas Sub-task Resolved Hairong Kuang
          14.
          		 Add a test for NameNode.getBlockLocations(..) to check read from un-closed file. Sub-task Resolved Tsz-wo Sze
          15.
          		 Introduce block committing logic during new block allocation and file close. Sub-task Resolved Konstantin Shvachko
          16.
          		 When opening a file for read, make the file length avaliable to client. Sub-task Resolved Tsz-wo Sze
          17.
          		 Extend Block report to include under-construction replicas Sub-task Resolved Konstantin Shvachko
          18.
          		 Datanode should serve up to visible length of a replica for read requests Sub-task Resolved Tsz-wo Sze
          19.
          		 Change block write protocol to support pipeline recovery Sub-task Resolved Hairong Kuang
          20.
          		 Create new functional test for a block report. Sub-task Closed Konstantin I Boudnik
          21.
          		 Allow client to get a new generation stamp from NameNode Sub-task Resolved Hairong Kuang
          22.
          		 Block report processing for append Sub-task Resolved Konstantin Shvachko
          23.
          		 Create functional tests for new design of the block report Sub-task Closed Konstantin I Boudnik
          24.
          		 Support replica recovery initialization in datanode Sub-task Resolved Tsz-wo Sze
          25.
          		 Client support pipeline recovery Sub-task Resolved Hairong Kuang
          26.
          		 Support replica update in datanode Sub-task Resolved Tsz-wo Sze
          27.
          		 SafeMode should count only complete blocks. Sub-task Resolved Konstantin Shvachko
          28.
          		 Support pipeline close and close recovery Sub-task Resolved Hairong Kuang
          29.
          		 Lease recovery, concurrency support. Sub-task Resolved Konstantin Shvachko
          30.
          		 Replace BlockInfo.isUnderConstruction() with isComplete() Sub-task Resolved Konstantin Shvachko
          31.
          		 Simplify the codes in the replica related classes Sub-task Resolved Unassigned
          32.
          		 Remove unused legacy protocol methods. Sub-task Resolved Konstantin Shvachko
          33.
          		 Block recovery for primary data-node Sub-task Resolved Konstantin Shvachko
          34.
          		 DFSClient cannot read all the available bytes Sub-task Resolved Tsz-wo Sze
          35.
          		 Remove deprecated methods from InterDatanodeProtocol. Sub-task Closed Konstantin Shvachko
          36.
          		 Data-node upgrade problem Sub-task Resolved Hairong Kuang
          37.
          		 Unnecessary info message from DFSClient Sub-task Resolved Hairong Kuang
          38.
          		 DFSIO for append Sub-task Closed Konstantin Shvachko
          39.
          		 TestFileAppend2 sometimes hangs Sub-task Resolved Hairong Kuang
          40.
          		 TestFileAppend3#TC7 sometimes hangs Sub-task Closed Hairong Kuang
          41.
          		 NPE in FSDataset.updateReplicaUnderRecovery(..) Sub-task Closed Konstantin Shvachko
          42.
          		 Create block recovery tests that handle errors Sub-task Closed Hairong Kuang

          Activity

            People

              hairong Hairong Kuang
              hairong Hairong Kuang
              Votes:
              5 Vote for this issue
              Watchers:
              54 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: