Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-21620 Problem in scan query when using more than one column prefix filter in some cases.
  3. HBASE-21734

Some optimization in FilterListWithOR

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 3.0.0-alpha-1, 2.2.0, 1.4.10, 2.1.3, 2.0.5
    • None
    • None
    • Reviewed
    • Hide
      After HBASE-21620, the filterListWithOR has been a bit slow because we need to merge each sub-filter's RC , while before HBASE-21620, we will skip many RC merging, but the logic was wrong. So here we choose another way to optimaze the performance: removing the KeyValueUtil#toNewKeyCell.
      Anoop Sam John suggested that the KeyValueUtil#toNewKeyCell can save some GC before because if we copy key part of cell into a single byte[], then the block the cell refering won't be refered by the filter list any more, the upper layer can GC the data block quickly. while after HBASE-21620, we will update the prevCellList for every encountered cell now, so the lifecycle of cell in prevCellList for FilterList will be quite shorter. so just use the cell ref for saving cpu.
      BTW, we removed all the arrays streams usage in filter list, because it's also quite time-consuming in our test.
      Show
      After HBASE-21620 , the filterListWithOR has been a bit slow because we need to merge each sub-filter's RC , while before HBASE-21620 , we will skip many RC merging, but the logic was wrong. So here we choose another way to optimaze the performance: removing the KeyValueUtil#toNewKeyCell. Anoop Sam John suggested that the KeyValueUtil#toNewKeyCell can save some GC before because if we copy key part of cell into a single byte[], then the block the cell refering won't be refered by the filter list any more, the upper layer can GC the data block quickly. while after HBASE-21620 , we will update the prevCellList for every encountered cell now, so the lifecycle of cell in prevCellList for FilterList will be quite shorter. so just use the cell ref for saving cpu. BTW, we removed all the arrays streams usage in filter list, because it's also quite time-consuming in our test.

    Description

      In HBASE-21620, KarthickRam and mohamed.meeran complaind that their performance of filter list has been degraded after that patch in here [1].

      I wrote a UT for this, and test under my host. It's true. I gussed there may be two reasons:
      1. the comparator.compare(nextKV, cell) > 0 StoreScanner;
      2. the filter list concated by OR will choose the minimal forward step among all sub-filters. in this patch, we have stricter restrictions on all sub filters, include those sub-filter whose has non-null RC returned in calculateReturnCodeByPrevCellAndRC (previously, we will skip to merge this sub-filter's rc, but it's wrong in some case), and merge all of the sub-filter's RC, this is also some time cost.

      The former one seems not the main problem, because the UT still cost ~ 3s even if I comment the compare. the second one has some impact indeed, because after i skip to merge the sub-filters's RC if calculateReturnCodeByPrevCellAndRC return a non-null rc, the UT cost ~ 1s, it's improvement but the logic is not wrong.

      1. https://issues.apache.org/jira/browse/HBASE-21620?focusedCommentId=16737100&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16737100

      Attachments

        1. perf-ut.patch
          3 kB
          Zheng Hu
        2. HBASE-21734.v1.patch
          3 kB
          Zheng Hu
        3. HBASE-21734.branch-1.v1.patch
          1 kB
          Zheng Hu
        4. columnkey.txt
          15.11 MB
          Zheng Hu

        Issue Links

          links to

          Web Link Review board

          Activity

            People

              openinx Zheng Hu
              openinx Zheng Hu
              Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: