Major Description
Daily Spark2 Structured streaming application built against the latest change on master started failing on 2022-07-23 with the error:
22/07/28 10:52:50 ERROR org.apache.hudi.HoodieStreamingSink: Micro batch id=2 threw following exception:
org.apache.spark.sql.catalyst.analysis.NoSuchDatabaseException: Database 'test' not found;
Actually the database exists (created by the Hudi). Spark3 application with the same code works fine.
Hudi options used for writing:
Map("hoodie.datasource.write.operation" -> "upsert", "hoodie.datasource.write.table.type" -> "MERGE_ON_READ", "hoodie.datasource.write.recordkey.field" -> s"$Meta.$Key", "hoodie.datasource.write.precombine.field" -> s"$Meta.$Version", "hoodie.datasource.write.payload.class" -> "com.walmart.udps.hudi.integration.HudiTestAvroPayload", "hoodie.compaction.payload.class" -> "com.walmart.udps.hudi.integration.HudiTestAvroPayload", "hoodie.datasource.write.partitionpath.field" -> TopLevelPartition, "hoodie.datasource.write.hive_style_partitioning" -> "true", "hoodie.datasource.hive_sync.enable" -> "true", "hoodie.datasource.hive_sync.mode" -> "hms", "hoodie.datasource.hive_sync.database" -> dbName(), "hoodie.datasource.hive_sync.partition_fields" -> TopLevelPartition, "hoodie.datasource.compaction.async.enable" -> "false" )
Cloud platform - Google Dataproc.
Spark version - 3.1.3.