Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-16961

FileSystem Quotas

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 2.0.0
    • None
    • None
    • Reviewed

    Description

      Umbrella issue for tracking the filesystem utilization of HBase data, defining quotas on that utilization, and enforcement when utilization exceeds the limits of the quota.

      At a high level: we can define quotas on tables and namespaces. Region size is computed by RegionServers and sent to the Master. The Master inspects the sizes of Regions, rolling up to table and namespace sizes. Defined quotas in the quota table are evaluated given the computed sizes, and, for those tables/namespaces violating the quota, RegionServers are informed to take some action to limit any further filesystem growth by that table/namespace.

      Discuss: https://lists.apache.org/thread.html/66a4b0c3725b5cbdd61dd6111c43847adaeef7b7da5f4cd045df30ef@%3Cdev.hbase.apache.org%3E
      Design Doc: http://home.apache.org/~elserj/hbase/FileSystemQuotasforApacheHBase.pdf or https://docs.google.com/document/d/1VtLWDkB2tpwc_zgCNPE1ulZOeecF-YA2FYSK3TSs_bw/edit?usp=sharing

      Attachments

        1. hbase-quota-test.sh
          2 kB
          Josh Elser

        Issue Links

          is related to

          Improvement - An improvement or enhancement to an existing feature or task. HBASE-17974 Concurrency concerns in transition of space quota state with large clusters

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Open
          relates to

          Umbrella - An overarching type made of sub-tasks HBASE-17748 Include HBase Snapshots in Space Quotas

          • Major - Major loss of function.
          • Resolved
          1.
          		 Build client Java API and client protobuf messages Sub-task Resolved Josh Elser
          2.
          		 Implement storage/retrieval of filesystem-use quotas into quota table Sub-task Resolved Josh Elser
          3.
          		 [Master] Collect table use information from RegionServers Sub-task Resolved Josh Elser
          4.
          		 [Master] Analyze table use reports and update quota violations Sub-task Resolved Josh Elser
          5.
          		 [Master] Inform RegionServers on size quota violations Sub-task Resolved Josh Elser
          6.
          		 [RegionServer] Compute region filesystem space use and report to Master Sub-task Resolved Josh Elser
          7.
          		 [RegionServer] Implement enforcement of quota violation policies Sub-task Resolved Josh Elser
          8.
          		 [Master] Report quotas and active violations on Master UI and JMX metrics Sub-task Resolved Josh Elser
          9.
          		 Update book for filesystem use quotas Sub-task Resolved Josh Elser
          10.
          		 [Shell] Support space quota get/set via the shell Sub-task Resolved Josh Elser
          11.
          		 Missing functionality to remove space quota Sub-task Resolved Josh Elser
          12.
          		 Expand on shell commands for detailed insight Sub-task Resolved Josh Elser
          13.
          		 Automatically delete quota when table is deleted Sub-task Resolved Josh Elser
          14.
          		 Avoid sending FSUtilization reports to master when quota support is not enabled Sub-task Resolved Josh Elser
          15.
          		 Table quota not taking precedence over namespace quota Sub-task Resolved Josh Elser
          16.
          		 SecureBulkLoadEndpoint not implemented Sub-task Resolved Josh Elser
          17.
          		 HRegionServer#reportRegionSizesForQuotas() should respond to UnsupportedOperationException Sub-task Resolved Ted Yu
          18.
          		 Expire region reports in the HMaster Sub-task Resolved Josh Elser
          19.
          		 Tweak chore delay/period defaults Sub-task Resolved Josh Elser
          20.
          		 Remove lingering "violation" in favor of the accurate "snapshot" Sub-task Resolved Josh Elser
          21.
          		 Commit Reviewboard comments from Vlad Sub-task Resolved Josh Elser
          22.
          		 Remove stability annotation from public audience-marked classes Sub-task Resolved Josh Elser
          23.
          		 Enabled Master observer to delete quotas on table deletion by default Sub-task Resolved Josh Elser
          24.
          		 Investigate hbase superuser permissions in the face of quota violation Sub-task Resolved Josh Elser
          25.
          		 Roll list_quota_violations into list_quota_snapshots Sub-task Resolved Josh Elser

          Activity

            People

              elserj Josh Elser
              elserj Josh Elser
              Votes:
              0 Vote for this issue
              Watchers:
              17 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: