Uploaded image for project: 'Apache Trafodion (Retired)'
  1. Apache Trafodion (Retired)
  2. TRAFODION-2606

Regionserver log on CDH 5.7 swamped with global.memstore.upperLimit deprecated messages when Trafodion under high transactional load

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Critical
    • Resolution: Unresolved
    • 2.2.0
    • None
    • dtm
    • None
    • Testing trafodion on AWS 4 node d2.2xlarge cluster with Trafodion 2.2 daily build, using CDH 5.7. Ran a version of the OrderEntry benchmark to produce workload
    • Patch, Important

    Description

      Was running a basic Trafodion checkout to validate a recently installed instance, and saw huge growth in the regionserver's log file. In under 5 hours, I was close to rolling a log, and after 10 hours, I was 122MB into the second log. On a different cluster, it was days before the log rolled. So I investigated further.

      At the top, I found the usual startup stuff, and messages as I loaded my test tables,

      Then came pages and pages and pages and pages of errors about a deprecated call:

      2017-05-03 19:52:18,858 WARN org.apache.hadoop.hbase.io.util.HeapMemorySizeUtil: hbase.regionserver.global.memstore.upperLimit is deprecated by hbase.regionserver.global.memstore.size

      How many in this 5 hour old instance?

      grep " hbase.regionserver.global.memstore.upperLimit is deprecated by hbase.regionserver.global.memstore.size" hbase-cmf-trafhbase-REGIONSERVER-glacier01.cluster.local.log.out | wc -l
      1084218
      [trafodion@glacier01 hbase]$

      In other words, a lot of extra energy is spent logging in the hbase regionserver. Over a million of them.

      The loads to set up my tables are not transactional, so I didn't see the same level of message generation.

      To reproduce the problem, run a transactional workload on HBase 1.2/CDH 5.7. I use a version of the order entry benchmark.

      Attachments

        Activity

          People

            Unassigned Unassigned
            carolp222 Carol Pearson
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated: