[PHOENIX-1226] Exception in Tracing - ASF JIRA

XML

Word

Printable

JSON

Details

Type: Bug
Status: Resolved
Priority: Major
Resolution: Fixed
Affects Version/s: 4.1.0
Fix Version/s: None
Component/s: None
Labels:
None
Environment:

0.98.5 hbase, 4.1.0 phoenix

Description

I was exposed to an exception in the tracing code, during my test setup of Phoenix in the following code:

58062 [defaultRpcServer.handler=2,queue=0,port=53950] WARN  org.apache.hadoop.ipc.RpcServer  - defaultRpcServer.handler=2,queue=0,port=53950: caught: java.lang.IllegalArgumentException: offset (0) + length (4) exceed the capacity of the array: 3
	at org.apache.hadoop.hbase.util.Bytes.explainWrongLengthOrOffset(Bytes.java:600)
	at org.apache.hadoop.hbase.util.Bytes.toInt(Bytes.java:749)
	at org.apache.hadoop.hbase.util.Bytes.toInt(Bytes.java:725)
	at org.apache.phoenix.trace.TracingCompat.readAnnotation(TracingCompat.java:56)
	at org.apache.phoenix.trace.TraceMetricSource.receiveSpan(TraceMetricSource.java:121)
	at org.cloudera.htrace.Tracer.deliver(Tracer.java:81)
	at org.cloudera.htrace.impl.MilliSpan.stop(MilliSpan.java:70)
	at org.cloudera.htrace.TraceScope.close(TraceScope.java:70)
	at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106)
	at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:114)
	at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94)
	at java.lang.Thread.run(Thread.java:744)

It is related to the following line of code where we interpret all KV annotation values as byte-wise integers here:

https://github.com/apache/phoenix/blob/v4.1.0/phoenix-hadoop-compat/src/main/java/org/apache/phoenix/trace/TracingCompat.java#L56

Here is where HBase is adding a non-integer KV annotation:

https://github.com/apache/hbase/blob/0.98.5/hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RequestContext.java#L105

The fix should be simple, but I am not aware of all the related issues in changing this.

cc jesse_yates, samarth.jain@gmail.com, giacomotaylor