Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-21029

Miscount of memstore's heap/offheap size if same cell was put

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 2.1.0, 2.0.1
    • 2.1.1, 2.0.2
    • None
    • None

    Description

      We are now using memstore.heapSize() + memstore.offheapSize() to decide whether a flush is needed. But, if a same cell was put again in memstore, only the memstore's dataSize will be increased, the heap/offheap size won't. We encountered the problem that a user was putting the same kv again and again, but the memstore won't flush since the heap size was not counted properly. The RS was killed by system since not enough memory in the end.

      Actually, if MSLAB is used, the heap/offheap will increase no matter the cell is added or not. IIRC, memstore's heap/offheap size should always bigger than data size. We introduced heap/offheap size besides data size in order to reflect memory footprint more precisely.

          // If there's already a same cell in the CellSet and we are using MSLAB, we must count in the
          // MSLAB allocation size as well, or else there will be memory leak (occupied heap size larger
          // than the counted number)
          if (succ || mslabUsed) {
            cellSize = getCellLength(cellToAdd);
          }
          // heap/offheap size is changed only if the cell is truly added in the cellSet
          long heapSize = heapSizeChange(cellToAdd, succ);
          long offHeapSize = offHeapSizeChange(cellToAdd, succ);
      

      Attachments

        1. HBASE-21029.branch-2.0.002.patch
          3 kB
          Allan Yang
        2. HBASE-21029.branch-2.0.001.patch
          3 kB
          Allan Yang

        Issue Links

          Activity

            People

              allan163 Allan Yang
              allan163 Allan Yang
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: