Details
-
Bug
-
Status: Open
-
Critical
-
Resolution: Unresolved
-
2.2.0
-
None
-
None
-
Testing trafodion on AWS 4 node d2.2xlarge cluster with Trafodion 2.2 daily build, using CDH 5.7. Ran a version of the OrderEntry benchmark to produce workload
-
Patch, Important
Description
Was running a basic Trafodion checkout to validate a recently installed instance, and saw huge growth in the regionserver's log file. In under 5 hours, I was close to rolling a log, and after 10 hours, I was 122MB into the second log. On a different cluster, it was days before the log rolled. So I investigated further.
At the top, I found the usual startup stuff, and messages as I loaded my test tables,
Then came pages and pages and pages and pages of errors about a deprecated call:
2017-05-03 19:52:18,858 WARN org.apache.hadoop.hbase.io.util.HeapMemorySizeUtil: hbase.regionserver.global.memstore.upperLimit is deprecated by hbase.regionserver.global.memstore.size
How many in this 5 hour old instance?
grep " hbase.regionserver.global.memstore.upperLimit is deprecated by hbase.regionserver.global.memstore.size" hbase-cmf-trafhbase-REGIONSERVER-glacier01.cluster.local.log.out | wc -l
1084218
[trafodion@glacier01 hbase]$
In other words, a lot of extra energy is spent logging in the hbase regionserver. Over a million of them.
The loads to set up my tables are not transactional, so I didn't see the same level of message generation.
To reproduce the problem, run a transactional workload on HBase 1.2/CDH 5.7. I use a version of the order entry benchmark.