Details
-
New Feature
-
Status: Resolved
-
Critical
-
Resolution: Fixed
-
Impala 3.3.0
-
ghx-label-4
Description
Can Impala read from and write to google cloud storage GCS like the way it does with amazon s3
I have tested the use case with S3, but when talking to GCS impala errors out with:
Query: create table gcs_impala2 (title string) location 'gs://mybucket-gcs/some_data/' ERROR: AnalysisException: null CAUSED BY: RuntimeException: java.lang.ClassNotFoundException: Class com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem not found CAUSED BY: ClassNotFoundException: Class com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem not found
On the same cluster i have Hive talking to GCS using the GCS connector jar provided by google form :
https://cloud.google.com/dataproc/docs/concepts/connectors/install-storage-connector
Also, HDFS reads and writes from/to GCS.
Made sure java version matches and appropriate values are in classpath.
Appreciate your time and effort.
Thanks
Attachments
Issue Links
- is related to
-
IMPALA-10561 Add support for spilling to GCS
- Reopened
-
IMPALA-10568 Enable file handle cache for GCS files
- Open
-
IMPALA-10562 TestGracefulShutdown::test_shutdown_idle failes in GCE instance
- Open
- links to