Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-6581

Write to single replica in memory

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 2.6.0
    • datanode, hdfs-client, namenode
    • None

    Description

      Per discussion with the community on HDFS-5851, we will implement writing to a single replica in DN memory via DataTransferProtocol.

      This avoids some of the issues with short-circuit writes, which we can revisit at a later time.

      Attachments

        1. Test-Plan-for-HDFS-6581-Memory-Storage.pdf
          179 kB
          Xiaoyu Yao
        2. Test-Plan-for-HDFS-6581-Memory-Storage.pdf
          202 kB
          Arpit Agarwal
        3. HDFSWriteableReplicasInMemory.pdf
          127 kB
          Arpit Agarwal
        4. HDFS-6581.merge.15.patch
          260 kB
          Arpit Agarwal
        5. HDFS-6581.merge.14.patch
          279 kB
          Arpit Agarwal
        6. HDFS-6581.merge.12.patch
          307 kB
          Arpit Agarwal
        7. HDFS-6581.merge.11.patch
          310 kB
          Arpit Agarwal
        8. HDFS-6581.merge.10.patch
          290 kB
          Arpit Agarwal
        9. HDFS-6581.merge.09.patch
          250 kB
          Arpit Agarwal
        10. HDFS-6581.merge.08.patch
          248 kB
          Arpit Agarwal
        11. HDFS-6581.merge.07.patch
          236 kB
          Arpit Agarwal
        12. HDFS-6581.merge.06.patch
          235 kB
          Arpit Agarwal
        13. HDFS-6581.merge.05.patch
          235 kB
          Arpit Agarwal
        14. HDFS-6581.merge.04.patch
          237 kB
          Arpit Agarwal
        15. HDFS-6581.merge.03.patch
          233 kB
          Arpit Agarwal
        16. HDFS-6581.merge.02.patch
          232 kB
          Arpit Agarwal
        17. HDFS-6581.merge.01.patch
          231 kB
          Arpit Agarwal

        Issue Links

          is related to

          Bug - A problem which impairs or prevents the functions of the product. HDFS-6912 SharedFileDescriptorFactory should not allocate sparse files

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Closed
          is required by

          New Feature - A new feature of the product, which has yet to be developed. HDFS-5851 Support memory as a storage medium

          • Major - Major loss of function.
          • Open
          relates to

          Improvement - An improvement or enhancement to an existing feature or task. HDFS-6919 Enforce a single limit for RAM disk usage and replicas cached via locking

          • Major - Major loss of function.
          • Open
          requires

          Bug - A problem which impairs or prevents the functions of the product. HDFS-6898 DN must reserve space for a full block when an RBW block is created

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. HDFS-6981 Fix DN upgrade with layout version change

          • Major - Major loss of function.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. HDFS-6482 Use block ID-based block layout on datanodes

          • Major - Major loss of function.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. HDFS-6800 Support Datanode layout changes with rolling upgrade

          • Major - Major loss of function.
          • Closed
          1.
          		 Initial prototype implementation for replicas in memory using tmpfs Sub-task Resolved Arpit Agarwal
          2.
          		 Add LazyPersist flag to FileStatus Sub-task Closed Arpit Agarwal
          3.
          		 The HttpFSFileSystem should also return the isLazyPersist flag in the FileStatus Sub-task Resolved Arpit Agarwal
          4.
          		 Add LazyPersist flag to INodeFile, save it in FsImage and edit logs Sub-task Resolved Arpit Agarwal
          5.
          		 Propagate LazyPersist flag to DNs via DataTransferProtocol Sub-task Closed Arpit Agarwal
          6.
          		 Add new RAM_DISK storage type Sub-task Closed Arpit Agarwal
          7.
          		 DataNode should attempt to place replicas on transient storage first if lazyPersist flag is received Sub-task Closed Arpit Agarwal
          8.
          		 DN support for saving replicas to persistent storage and evicting in-memory replicas Sub-task Closed Arpit Agarwal
          9.
          		 Add unit tests Sub-task Closed Arpit Agarwal
          10.
          		 NN periodically unlinks lazy persist files with missing replicas from namespace Sub-task Closed Arpit Agarwal
          11.
          		 'hdfs put' command should accept lazyPersist flag for testing Sub-task Closed Arpit Agarwal
          12.
          		 Bugfix in LazyWriter, fix test case and some refactoring Sub-task Closed Arpit Agarwal
          13.
          		 Move lazily persisted replicas to finalized directory on DN startup Sub-task Closed Arpit Agarwal
          14.
          		 Add Additional unit tests for HDFS-6581 Sub-task Closed Xiaoyu Yao
          15.
          		 Improve replica eviction from RAM disk Sub-task Closed Arpit Agarwal
          16.
          		 Delete all copies when a block is deleted from the block space Sub-task Closed Arpit Agarwal
          17.
          		 Notify NN of evicted block before deleting it from RAM disk Sub-task Closed Arpit Agarwal
          18.
          		 Directory scanner should correctly reconcile blocks on RAM disk Sub-task Closed Arpit Agarwal
          19.
          		 LazyWriter#evictBlocks misses a null check for replicaState Sub-task Closed Xiaoyu Yao
          20.
          		 Fix unit test failures in HDFS-6581 branch Sub-task Closed Xiaoyu Yao
          21.
          		 Few more unit test fixes for HDFS-6581 Sub-task Closed Arpit Agarwal
          22.
          		 Fix finalize and upgrade unit test failures Sub-task Closed Arpit Agarwal
          23.
          		 FsDatasetImpl#copyBlockFiles debug log can be improved Sub-task Closed Xiaoyu Yao
          24.
          		 Add forwarding constructor for INodeFile for existing callers Sub-task Closed Arpit Agarwal
          25.
          		 Updated editsStored and editsStored.xml to bump layout version and add LazyPersist flag Sub-task Resolved Xiaoyu Yao
          26.
          		 Fix unit test failures in SimulatedFsDataset Sub-task Closed Arpit Agarwal
          27.
          		 Make eviction scheme pluggable Sub-task Closed Arpit Agarwal
          28.
          		 Add unit test for evict/delete RAM_DISK block with open handle Sub-task Closed Xiaoyu Yao
          29.
          		 Fix findbugs warnings in the HDFS-6581 branch Sub-task Resolved Arpit Agarwal
          30.
          		 Balancer and Mover tools should ignore replicas on RAM_DISK Sub-task Closed Xiaoyu Yao
          31.
          		 Fix findbugs warnings in HDFS-6581 branch Sub-task Closed Tsz-wo Sze
          32.
          		 Fix findbugs warnings in RamDiskReplicaTracker Sub-task Closed Tsz-wo Sze
          33.
          		 Integrate HDFS-6581 memory usage with HDFS-4949 Sub-task Resolved Colin McCabe
          34.
          		 Bugfix in createLocatedFileStatus caused by bad merge Sub-task Closed Arpit Agarwal
          35.
          		 Fix incorrect layout version caused by bad merge Sub-task Resolved Arpit Agarwal
          36.
          		 Use block storage policy to set lazy persist preference Sub-task Closed Arpit Agarwal
          37.
          		 Metrics to track usage of memory for writes Sub-task Closed Xiaoyu Yao
          38.
          		 Fix Jenkins failures in HDFS-6581 branch Sub-task Closed Arpit Agarwal
          39.
          		 Improve HDFS-6581 eviction configuration Sub-task Closed Xiaoyu Yao
          40.
          		 Use unbuffered writes when persisting in-memory replicas Sub-task Closed Xiaoyu Yao
          41.
          		 LazyWriter should use either async IO or one thread per physical disk Sub-task Closed Xiaoyu Yao
          42.
          		 Move checksum computation off the hot path when writing to RAM disk Sub-task Closed Chris Nauroth
          43.
          		 Persist in-memory replicas with appropriate unbuffered copy API on POSIX and Windows Sub-task Closed Xiaoyu Yao
          44.
          		 Feature documentation for HDFS-6581 Sub-task Closed Arpit Agarwal
          45.
          		 DN should ignore lazyPersist hint if the writer is not local Sub-task Resolved Arpit Agarwal
          46.
          		 Implement a 2Q eviction strategy for HDFS-6581 Sub-task Patch Available Colin McCabe
          47.
          		 Skip checksum verification when reading from memory Sub-task Open Unassigned
          48.
          		 Investigate eviction from RAM disk via file truncation Sub-task Open Unassigned
          49.
          		 DFSOutputStream should query policies from NameNode Sub-task Open Arpit Agarwal
          50.
          		 Support using ramfs partitions on Linux Sub-task Open Arpit Agarwal

          Activity

            People

              arp Arpit Agarwal
              arp Arpit Agarwal
              Votes:
              0 Vote for this issue
              Watchers:
              42 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: