Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-24578

[WAL] Add a parameter to config RingBufferEventHandler's SyncFuture count

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 1.4.13, 2.2.5
    • 3.0.0-alpha-1, 2.3.1, 1.7.0, 2.2.6
    • wal
    • None
    • Reviewed
    • Hide
      Introduce a new parameter "hbase.regionserver.wal.sync.batch.count" to control the wal sync batch size which is equals to "hbase.regionserver.handler.count" by default. It should work well if you use default wal provider---one wal per regionserver. But if you use read/write separated handlers, you can set "hbase.regionserver.wal.sync.batch.count" to the number of write handlers. And if you use wal-per-groups or wal-per-region, you can consider lower "hbase.regionserver.wal.sync.batch.count", the default number will be too big and consume more memories as the number of wals grows.
      Show
      Introduce a new parameter "hbase.regionserver.wal.sync.batch.count" to control the wal sync batch size which is equals to "hbase.regionserver.handler.count" by default. It should work well if you use default wal provider---one wal per regionserver. But if you use read/write separated handlers, you can set "hbase.regionserver.wal.sync.batch.count" to the number of write handlers. And if you use wal-per-groups or wal-per-region, you can consider lower "hbase.regionserver.wal.sync.batch.count", the default number will be too big and consume more memories as the number of wals grows.

    Description

      The current value of RingBufferEventHandler's handler is the value of hbase.regionserver.handler.count, which works good in default wal provider — one WAL per regionserver.

      When trying to use WAL group provider, either by group or wal per region, the default value is bad. If rs has 100 regions and wal per region strategy is used, then rs will allocate 100 * SyncFuture[$hbase.regionserver.handler.count] array

      int maxHandlersCount = conf.getInt(HConstants.REGION_SERVER_HANDLER_COUNT, 200);
    this.ringBufferEventHandler = new RingBufferEventHandler(
        conf.getInt("hbase.regionserver.hlog.syncer.count", 5), maxHandlersCount); 
...

----

RingBufferEventHandler(final int syncRunnerCount, final int maxHandlersCount) {
      this.syncFutures = new SyncFuture[maxHandlersCount];
      ...
 }

      Attachments

        Activity

          People

            wenfeiyi666 wenfeiyi666
            reidchan Reid Chan
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: