Uploaded image for project: 'Apache Hudi'
  1. Apache Hudi
  2. HUDI-3107

Fix HiveSyncTool drop partitions using JDBC

    XMLWordPrintableJSON

Details

    • 1

    Description

      ```
      org.apache.hudi.exception.HoodieException: Unable to delete table partitions in /Users/yuezhang/tmp/hudiAfTable/forecast_agg
      at org.apache.hudi.utilities.HoodieDropPartitionsTool.run(HoodieDropPartitionsTool.java:240)
      at org.apache.hudi.utilities.HoodieDropPartitionsTool.main(HoodieDropPartitionsTool.java:212)
      at HoodieDropPartitionsToolTest.main(HoodieDropPartitionsToolTest.java:31)
      Caused by: org.apache.hudi.exception.HoodieException: Got runtime exception when hive syncing forecast_agg
      at org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:119)
      at org.apache.hudi.utilities.HoodieDropPartitionsTool.syncHive(HoodieDropPartitionsTool.java:404)
      at org.apache.hudi.utilities.HoodieDropPartitionsTool.syncToHiveIfNecessary(HoodieDropPartitionsTool.java:270)
      at org.apache.hudi.utilities.HoodieDropPartitionsTool.doDeleteTablePartitionsEager(HoodieDropPartitionsTool.java:252)
      at org.apache.hudi.utilities.HoodieDropPartitionsTool.run(HoodieDropPartitionsTool.java:230)
      ... 2 more
      Caused by: org.apache.hudi.hive.HoodieHiveSyncException: Failed to sync partitions for table forecast_agg
      at org.apache.hudi.hive.HiveSyncTool.syncPartitions(HiveSyncTool.java:368)
      at org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:202)
      at org.apache.hudi.hive.HiveSyncTool.doSync(HiveSyncTool.java:130)
      at org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:116)
      ... 6 more
      Caused by: org.apache.hudi.hive.HoodieHiveSyncException: Failed in executing SQL ALTER TABLE `forecast_agg` DROP PARTITION (20210623/0/20210623)
      at org.apache.hudi.hive.ddl.JDBCExecutor.runSQL(JDBCExecutor.java:64)
      at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184)
      at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
      at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382)
      at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
      at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
      at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151)
      at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174)
      at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
      at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418)
      at org.apache.hudi.hive.ddl.JDBCExecutor.dropPartitionsToTable(JDBCExecutor.java:149)
      at org.apache.hudi.hive.HoodieHiveClient.dropPartitionsToTable(HoodieHiveClient.java:130)
      at org.apache.hudi.hive.HiveSyncTool.syncPartitions(HiveSyncTool.java:363)
      ... 9 more
      Caused by: org.apache.hive.service.cli.HiveSQLException: Error while compiling statement: FAILED: ParseException line 1:43 cannot recognize input near '20210623' '/' '0' in drop partition statement
      at org.apache.hive.jdbc.Utils.verifySuccess(Utils.java:256)
      at org.apache.hive.jdbc.Utils.verifySuccessWithInfo(Utils.java:242)
      at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:254)
      at org.apache.hudi.hive.ddl.JDBCExecutor.runSQL(JDBCExecutor.java:62)
      ... 21 more
      Caused by: org.apache.hive.service.cli.HiveSQLException: Error while compiling statement: FAILED: ParseException line 1:43 cannot recognize input near '20210623' '/' '0' in drop partition statement
      at org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:380)
      at org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:206)
      at org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:290)
      at org.apache.hive.service.cli.operation.Operation.run(Operation.java:320)
      at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:530)
      at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:517)
      at org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:310)
      at org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:530)
      at org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1437)
      at org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1422)
      at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
      at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
      at org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:56)
      at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      at java.lang.Thread.run(Thread.java:748)
      Caused by: java.lang.RuntimeException: org.apache.hadoop.hive.ql.parse.ParseException:line 1:43 cannot recognize input near '20210623' '/' '0' in drop partition statement
      at org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:211)
      at org.apache.hadoop.hive.ql.parse.ParseUtils.parse(ParseUtils.java:77)
      at org.apache.hadoop.hive.ql.parse.ParseUtils.parse(ParseUtils.java:70)
      at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:468)
      at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1317)
      at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1295)
      at org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:204)
      ... 15 more
      ```

      Attachments

        Issue Links

          Activity

            People

              zhangyue19921010 Yue Zhang
              zhangyue19921010 Yue Zhang
              Shiyan Xu
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: