Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-13553 RBF: Support global quota
  3. HDFS-12934

RBF: Federation supports global quota

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.0.0
    • 3.1.0, 2.10.0
    • None
    • Reviewed

    Description

      Now federation doesn't support set the global quota for each folder. Currently the quota will be applied for each subcluster under the specified folder via RPC call.
      It will be very useful for users that federation can support setting global quota and exposing the command of this.

      In a federated environment, a folder can be spread across multiple subclusters. For this reason, we plan to solve this by following way:

      1. Set global quota across each subcluster. We don't allow each subcluster can exceed maximun quota value.
      2. We need to construct one <Path, QuotaUsage> cache map for storing the sum quota usage of these subclusters under federation folder. Every time we want to do WRITE operation under specified folder, we will get its quota usage from cache and verify its quota. If quota exceeded, throw exception, otherwise update its quota usage in cache when finishing operations.

      The quota will be set to mount table and as a new field in mount table. The set/unset command will be like:

       hdfs dfsrouteradmin -setQuota -ns <nsQuota> -ss <ssQuota> <mount table>
 hdfs dfsrouteradmin -clrQuota  <mount table>

      Attachments

        1. RBF support global quota.pdf
          248 kB
          Yiqun Lin
        2. HDFS-12934.001.patch
          52 kB
          Yiqun Lin
        3. HDFS-12934.002.patch
          53 kB
          Yiqun Lin
        4. HDFS-12934.003.patch
          60 kB
          Yiqun Lin
        5. HDFS-12934.004.patch
          65 kB
          Yiqun Lin
        6. HDFS-12934.005.patch
          67 kB
          Yiqun Lin
        7. HDFS-12934.006.patch
          72 kB
          Yiqun Lin
        8. HDFS-12934.007.patch
          76 kB
          Yiqun Lin
        9. HDFS-12934.008.patch
          78 kB
          Yiqun Lin
        10. HDFS-12934-branch-2.001.patch
          78 kB
          Yiqun Lin
        11. HDFS-12934-branch-2.002.patch
          78 kB
          Yiqun Lin
        12. HDFS-12919-branch-3.001.patch
          48 kB
          Íñigo Goiri

        Issue Links

          contains

          Sub-task - The sub-task of the issue HDFS-12972 RBF: Display mount table quota info in Web UI and admin command

          • Major - Major loss of function.
          • Resolved

          Sub-task - The sub-task of the issue HDFS-12973 RBF: Document global quota supporting in federation

          • Major - Major loss of function.
          • Resolved
          relates to

          Sub-task - The sub-task of the issue HDFS-13253 RBF: Quota management incorrect parent-child relationship judgement

          • Major - Major loss of function.
          • Resolved

          Sub-task - The sub-task of the issue HDFS-13307 RBF: Improve the use of setQuota command

          • Major - Major loss of function.
          • Resolved

          Sub-task - The sub-task of the issue HDFS-13346 RBF: Fix synchronization of router quota and nameservice quota

          • Major - Major loss of function.
          • Resolved

          Sub-task - The sub-task of the issue HDFS-13380 RBF: mv/rm fail after the directory exceeded the quota limit

          • Major - Major loss of function.
          • Resolved

          Sub-task - The sub-task of the issue HDFS-13528 RBF: If a directory exceeds quota limit then quota usage is not refreshed for other mount entries

          • Major - Major loss of function.
          • Resolved

          Activity

            People

              linyiqun Yiqun Lin
              linyiqun Yiqun Lin
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: