Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-6423

Writes should not block reads on blocking updates to memstores

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 0.94.4, 0.95.0
    • None
    • None
    • Reviewed
    • Hide
      Added new configuration parameters to adjust the wait time in getting a lock before acting on a region:

      hbase.busy.wait.multiplier.max (default 2)
      hbase.rowlock.wait.duration (same as the default RPC timeout time)

      For reading, it waits at most "hbase.rowlock.wait.duration" in getting a lock.
      For writing, it waits at most "hbase.rowlock.wait.duration" * min ( #rows affected, "
      hbase.busy.wait.multiplier.max"). However, it waits at most the server side call purge timeout time.

      Show
      Added new configuration parameters to adjust the wait time in getting a lock before acting on a region: hbase.busy.wait.multiplier.max (default 2) hbase.rowlock.wait.duration (same as the default RPC timeout time) For reading, it waits at most "hbase.rowlock.wait.duration" in getting a lock. For writing, it waits at most "hbase.rowlock.wait.duration" * min ( #rows affected, " hbase.busy.wait.multiplier.max"). However, it waits at most the server side call purge timeout time.

    Description

      We have a big data use case where we turn off WAL and have a ton of reads and writes. We found that:

      1. flushing a memstore takes a while (GZIP compression)
      2. incoming writes cause the new memstore to grow in an unbounded fashion
      3. this triggers blocking memstore updates
      4. in turn, this causes all the RPC handler threads to block on writes to that memstore
      5. we are not able to read during this time as RPC handlers are blocked

      At a higher level, we should not hold up the RPC threads while blocking updates, and we should build in some sort of rate control.

      Attachments

        1. 0.94-6423_v4.patch
          44 kB
          Jimmy Xiang
        2. 0.94-6423.patch
          43 kB
          Jimmy Xiang
        3. 6423.addendum
          0.6 kB
          Ted Yu
        4. trunk-6423_v2.1.patch
          39 kB
          Jimmy Xiang
        5. trunk-6423_v2.patch
          40 kB
          Jimmy Xiang
        6. trunk-6423_v3.2.patch
          39 kB
          Jimmy Xiang
        7. trunk-6423_v3.3.patch
          40 kB
          Jimmy Xiang
        8. trunk-6423_v3.4.patch
          41 kB
          Jimmy Xiang
        9. trunk-6423_v4.patch
          42 kB
          Jimmy Xiang
        10. trunk-6423.patch
          40 kB
          Jimmy Xiang

        Issue Links

          is related to

          Bug - A problem which impairs or prevents the functions of the product. HBASE-7261 batchMutate/put release the same lock twice

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. HBASE-7376 Acquiring readLock does not apply timeout in HRegion#flushcache

          • Major - Major loss of function.
          • Closed
          relates to

          New Feature - A new feature of the product, which has yet to be developed. HBASE-5162 Basic client pushback mechanism

          • Major - Major loss of function.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. HBASE-5333 Introduce Memstore "backpressure" for writes

          • Major - Major loss of function.
          • Closed

          Brainstorming - A place to record back and forth on notions not yet formed enough to make a 'New Feature' or 'Task' HBASE-2981 Improve consistency of performance during heavy load

          • Major - Major loss of function.
          • Closed

          Activity

            People

              jxiang Jimmy Xiang
              karthik.ranga Karthik Ranganathan
              Votes:
              0 Vote for this issue
              Watchers:
              16 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: