Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-4750 Support NFSv3 interface to HDFS
  3. HDFS-5364

Add OpenFileCtx cache

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 2.3.0
    • nfs
    • None
    • Reviewed

    Description

      NFS gateway can run out of memory when the stream timeout is set to a relatively long period(e.g., >1 minute) and user uploads thousands of files in parallel. Each stream DFSClient creates a DataStreamer thread, and will eventually run out of memory by creating too many threads.

      NFS gateway should have a OpenFileCtx cache to limit the total opened files.

      Attachments

        1. HDFS-5364.001.patch
          19 kB
          Brandon Li
        2. HDFS-5364.002.patch
          25 kB
          Brandon Li
        3. HDFS-5364.003.patch
          30 kB
          Brandon Li
        4. HDFS-5364.004.patch
          37 kB
          Brandon Li
        5. HDFS-5364.005.patch
          37 kB
          Brandon Li
        6. HDFS-5364.006.patch
          38 kB
          Brandon Li
        7. HDFS-5364.007.patch
          41 kB
          Brandon Li
        8. HDFS-5364.008.patch
          43 kB
          Brandon Li
        9. HDFS-5364.009.patch
          43 kB
          Brandon Li

        Issue Links

          is duplicated by

          Sub-task - The sub-task of the issue HDFS-5172 Handle race condition for writes

          • Major - Major loss of function.
          • Resolved

          Activity

            People

              brandonli Brandon Li
              brandonli Brandon Li
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: