Details
-
Sub-task
-
Status: Resolved
-
Major
-
Resolution: Duplicate
-
None
-
None
-
None
Description
It seems unlikely that Spark community will provide MRMiniCluster equivalent (SPARK-3691), and Spark local-cluster was the recommendation. Latest research shows that Spark local-cluster works with Hive. Therefore, for now, we use Spark local-cluster (instead of current local) for our precommit test.
It's previous belived (HIVE-7382) that a Spark installation is required and SPARK_HOME env variable needs to set. Since Spark pulls in Spark's assembly jar, it's believed now we only need a few script from Spark installation instead.
Attachments
Issue Links
- duplicates
-
HIVE-8836 Enable automatic tests with remote spark client [Spark Branch]
- Resolved