Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-8329

Limit compaction speed

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 1.1.0, 0.98.11, 2.0.0
    • Compaction
    • None
    • Reviewed
    • Hide
      Adds compaction throughput limit mechanism(the word "throttle" is already used when choosing compaction thread pool, so use a different word here to avoid ambiguity). Default is org.apache.hadoop.hbase.regionserver.compactions.PressureAwareCompactionThroughputController, will limit throughput as follow:
      1. In off peak hours, use a fixed limitation "hbase.hstore.compaction.throughput.offpeak" (default is Long.MAX_VALUE which means no limitation).
      2. In normal hours, the limitation is tuned between "hbase.hstore.compaction.throughput.lower.bound"(default 10MB/sec) and "hbase.hstore.compaction.throughput.higher.bound"(default 20MB/sec), using the formula "lower + (higer - lower) * param" where param is in range [0.0, 1.0] and calculate based on store files count on this regionserver.
      3. If some stores have too many store files(storefilesCount > blockingFileCount), then there is no limitation no matter peak or off peak.
      You can set "hbase.regionserver.throughput.controller" to org.apache.hadoop.hbase.regionserver.throttle.NoLimitThroughputController to disable throughput controlling.
      And we have implemented ConfigurationObserver which means you can change all configurations above and do not need to restart cluster.

      The throttle is on by default in hbase-2.0.0. There is no limit in hbase-1.x.
      Show
      Adds compaction throughput limit mechanism(the word "throttle" is already used when choosing compaction thread pool, so use a different word here to avoid ambiguity). Default is org.apache.hadoop.hbase.regionserver.compactions.PressureAwareCompactionThroughputController, will limit throughput as follow: 1. In off peak hours, use a fixed limitation "hbase.hstore.compaction.throughput.offpeak" (default is Long.MAX_VALUE which means no limitation). 2. In normal hours, the limitation is tuned between "hbase.hstore.compaction.throughput.lower.bound"(default 10MB/sec) and "hbase.hstore.compaction.throughput.higher.bound"(default 20MB/sec), using the formula "lower + (higer - lower) * param" where param is in range [0.0, 1.0] and calculate based on store files count on this regionserver. 3. If some stores have too many store files(storefilesCount > blockingFileCount), then there is no limitation no matter peak or off peak. You can set "hbase.regionserver.throughput.controller" to org.apache.hadoop.hbase.regionserver.throttle.NoLimitThroughputController to disable throughput controlling. And we have implemented ConfigurationObserver which means you can change all configurations above and do not need to restart cluster. The throttle is on by default in hbase-2.0.0. There is no limit in hbase-1.x.

    Description

      There is no speed or resource limit for compaction,I think we should add this feature especially when request burst.

      Attachments

        1. HBASE-8329-trunk.patch
          6 kB
          Lijin Bin
        2. HBASE-8329-2-trunk.patch
          7 kB
          Lijin Bin
        3. HBASE-8329-3-trunk.patch
          10 kB
          Lijin Bin
        4. HBASE-8329-4-trunk.patch
          10 kB
          Lijin Bin
        5. HBASE-8329-5-trunk.patch
          10 kB
          Lijin Bin
        6. HBASE-8329-6-trunk.patch
          10 kB
          Lijin Bin
        7. HBASE-8329-7-trunk.patch
          14 kB
          Lijin Bin
        8. HBASE-8329-8-trunk.patch
          21 kB
          Lijin Bin
        9. HBASE-8329-9-trunk.patch
          22 kB
          Sergey Shelukhin
        10. HBASE-8329-10.patch
          29 kB
          Duo Zhang
        11. HBASE-8329-11.patch
          29 kB
          Duo Zhang
        12. HBASE-8329-12.patch
          35 kB
          Duo Zhang
        13. HBASE-8329_13.patch
          83 kB
          Duo Zhang
        14. HBASE-8329_14.patch
          102 kB
          Duo Zhang
        15. HBASE-8329_15.patch
          102 kB
          Duo Zhang
        16. HBASE-8329_16.patch
          104 kB
          Duo Zhang
        17. HBASE-8329_17.patch
          104 kB
          Duo Zhang
        18. HBASE-8329-branch-1.patch
          104 kB
          Duo Zhang
        19. HBASE-8329-0.98.patch
          107 kB
          Duo Zhang
        20. HBASE-8329-0.98-addendum.patch
          7 kB
          Andrew Kyle Purtell

        Issue Links

          Activity

            People

              zhangduo Duo Zhang
              binlijin Lijin Bin
              Votes:
              0 Vote for this issue
              Watchers:
              30 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: