Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-5162

Basic client pushback mechanism

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 0.92.0
    • 2.0.0
    • None
    • None

    Description

      The current blocking we do when we are close to some limits (memstores over the multiplier factor, too many store files, global memstore memory) is bad, too coarse and confusing. After hitting HBASE-5161, it really becomes obvious that we need something better.

      I did a little brainstorm with Stack, we came up quickly with two solutions:

      • Send some exception to the client, like OverloadedException, that's thrown when some situation happens like getting past the low memory barrier. It would be thrown when the client gets a handler and does some check while putting or deleting. The client would treat this a retryable exception but ideally wouldn't check .META. for a new location. It could be fancy and have multiple levels of pushback, like send the exception to 25% of the clients, and then go up if the situation persists. Should be "easy" to implement but we'll be using a lot more IO to send the payload over and over again (but at least it wouldn't sit in the RS's memory).
      • Send a message alongside a successful put or delete to tell the client to slow down a little, this way we don't have to do back and forth with the payload between the client and the server. It's a cleaner (I think) but more involved solution.

      In every case the RS should do very obvious things to notify the operators of this situation, through logs, web UI, metrics, etc.

      Other ideas?

      Attachments

        1. java_HBASE-5162.patch
          39 kB
          Jesse Yates
        2. hbase-5162-trunk-v0.patch
          143 kB
          Jesse Yates
        3. hbase-5162-trunk-v1.patch
          95 kB
          Jesse Yates
        4. hbase-5162-trunk-v2.patch
          101 kB
          Jesse Yates
        5. hbase-5162-trunk-v3.patch
          100 kB
          Jesse Yates
        6. hbase-5162-trunk-v4.patch
          100 kB
          Jesse Yates
        7. hbase-5162-trunk-v5.patch
          107 kB
          Jesse Yates
        8. hbase-5162-trunk-v6.patch
          109 kB
          Jesse Yates
        9. hbase-5162-trunk-v7.patch
          109 kB
          Jesse Yates
        10. hbase-5162-trunk-v8.patch
          129 kB
          Jesse Yates
        11. hbase-5162-trunk-v10.patch
          132 kB
          Jesse Yates
        12. hbase-5162-trunk-v11.patch
          133 kB
          Jesse Yates
        13. hbase-5162-trunk-v12-committed.patch
          133 kB
          Jesse Yates
        14. hbase-5162-branch-1-v0.patch
          132 kB
          Jesse Yates
        15. hbase-5162-trunk-addendum.patch
          4 kB
          Jesse Yates
        16. 5162-addendum2.txt
          0.9 kB
          Ted Yu

        Issue Links

          is duplicated by

          Improvement - An improvement or enhancement to an existing feature or task. HBASE-5333 Introduce Memstore "backpressure" for writes

          • Major - Major loss of function.
          • Closed
          is related to

          Bug - A problem which impairs or prevents the functions of the product. HBASE-6423 Writes should not block reads on blocking updates to memstores

          • Major - Major loss of function.
          • Closed

          New Feature - A new feature of the product, which has yet to be developed. HBASE-12731 Heap occupancy based client pushback

          • Major - Major loss of function.
          • Closed

          New Feature - A new feature of the product, which has yet to be developed. HBASE-12911 Client-side metrics

          • Major - Major loss of function.
          • Closed

          Brainstorming - A place to record back and forth on notions not yet formed enough to make a 'New Feature' or 'Task' HBASE-2981 Improve consistency of performance during heavy load

          • Major - Major loss of function.
          • Closed
          relates to

          Sub-task - The sub-task of the issue HBASE-22267 Implement client push back for async client

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. HBASE-12703 Cleanup TestClientPushback for repeatability

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Closed

          New Feature - A new feature of the product, which has yet to be developed. HBASE-12217 System load average based client pushback

          • Major - Major loss of function.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. HBASE-12840 Improve unit test coverage of the client pushback mechanism

          • Major - Major loss of function.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. HBASE-12841 ClientBackoffPolicies should support immediate rejection of submitted ops

          • Major - Major loss of function.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. HBASE-12986 Compaction pressure based client pushback

          • Major - Major loss of function.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. HBASE-14693 Add client-side metrics for received pushback signals

          • Major - Major loss of function.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. HBASE-14756 Break out ClientBackoffPolicy factors into configurable and composable components

          • Major - Major loss of function.
          • Closed

          Task - A task that needs to be done. HBASE-12906 Document the new load pushback mechanism and available policies

          • Major - Major loss of function.
          • Closed

          Umbrella - An overarching type made of sub-tasks HBASE-12702 Backport client pushback

          • Major - Major loss of function.
          • Closed

          Activity

            People

              jesse_yates Jesse Yates
              jdcryans Jean-Daniel Cryans
              Votes:
              0 Vote for this issue
              Watchers:
              30 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: