Details
-
Sub-task
-
Status: Closed
-
Critical
-
Resolution: Not A Bug
-
3.0.0
-
None
-
None
Description
Originally reported at https://github.com/apache/spark/pull/26619#issuecomment-583802901
Right now, Hive 2.3 profile pulls in regular orc, which depends on hive-storage-api. However, hive-storage-api and hive-common have the following common class files
org/apache/hadoop/hive/common/ValidReadTxnList.class
org/apache/hadoop/hive/common/ValidTxnList.class
org/apache/hadoop/hive/common/ValidTxnList$RangeResponse.class
For example, https://github.com/apache/hive/blob/rel/storage-release-2.6.0/storage-api/src/java/org/apache/hadoop/hive/common/ValidReadTxnList.java (pulled in by orc 1.5.8) and https://github.com/apache/hive/blob/rel/release-2.3.6/common/src/java/org/apache/hadoop/hive/common/ValidReadTxnList.java (from hive-common 2.3.6) both are in the classpath and they are different. Having both versions in the classpath can cause unexpected behavior due to classloading order. We should still use orc-nohive, which has hive-storage-api shaded.
Attachments
Issue Links
- is caused by
-
SPARK-27176 Upgrade hadoop-3's built-in Hive maven dependencies to 2.3.4
- Resolved
- links to