Details
-
Bug
-
Status: Patch Available
-
Major
-
Resolution: Unresolved
-
0.14.0, 1.2.2, 2.3.0
-
None
-
None
Description
LazySimpleSerDe.LazyMap doesn't handle the condition of when map key is another column value.
eg:
CREATE TABLE `demo`(
`create_timestamp` bigint,
`ctime` string,
`event_data` map<string,string>
)
ROW FORMAT SERDE
'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe'
STORED AS INPUTFORMAT
'org.apache.hadoop.mapred.TextInputFormat'
OUTPUTFORMAT
'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'
add data
INSERT INTO TABLE demo
select
1511192567909,
'1511192567909',
map('1511192567909', '05020918127a82816437ba9edac17fca42601');
Do query:
SELECT
create_timestamp,
ctime,
event_data,
event_data[create_timestamp],
event_data[ctime]
FROM demo;
RES:
create_timestamp | ctime | event_data | _c3 | _c4 |
--------------------------------------------------------------------------------------------------------------------------------------+
1511192567909 | 1511192567909 | {"1511192567909":"05020918127a82816437ba9edac17fca42601"} | 05020918127a82816437ba9edac17fca42601 | NULL |
event_data[ctime] get value NULL
_______________________________________
In LazyMap.getMapValueElement method.
// Some comments here public Object getMapValueElement(Object key) { if (!parsed) { parse(); } // search for the key for (int i = 0; i < mapSize; i++) { LazyPrimitive<?, ?> lazyKeyI = uncheckedGetKey(i); if (lazyKeyI == null) { continue; } // getWritableObject() will convert LazyPrimitive to actual primitive // writable objects. Object keyI = lazyKeyI.getWritableObject(); if (keyI == null) { continue; } if (keyI.equals(key)) { // Got a match, return the value return uncheckedGetValue(i); } } return null; }
When the key is a cloumn and don't need convert, the key will be a LazyPrimitive object.
keyI = lazyKeyI.getWritableObject() is actual primitive writable objects.
keyI.equals(key) will be false.