Uploaded image for project: 'Kylin'
  1. Kylin
  2. KYLIN-4107

StorageCleanupJob fails to delete Hive tables with "Argument list too long" error

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • v2.6.2
    • v3.0.0-beta
    • Storage - HBase
    • None
    • CentOS 7.6, HDP 2.6.5, Kylin 2.6.3

    Description

      On a system with multiple Kylin developers that experiment with cube design and (re)build/drop cube segments often intermediate Hive tables and HBase left over tables accumulate very quickly.

      After a certain point storage cleanup cannot be executed using suggested method:
      ${KYLIN_HOME}/bin/kylin.sh org.apache.kylin.tool.StorageCleanupJob --delete true

      Apparently, storage cleanup job creates a single shell command to drop all Hive tables, which fails to execute because command line is just too long. For example:

      2019-07-23 17:47:31,611 ERROR [main] job.StorageCleanupJob:377 : Error during deleting Hive tables
      java.io.IOException: Cannot run program "/bin/bash": error=7, Argument list too long
      at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048)
      at org.apache.kylin.common.util.CliCommandExecutor.runNativeCommand(CliCommandExecutor.java:133)
      at org.apache.kylin.common.util.CliCommandExecutor.execute(CliCommandExecutor.java:89)
      at org.apache.kylin.common.util.CliCommandExecutor.execute(CliCommandExecutor.java:83)
      at org.apache.kylin.rest.job.StorageCleanupJob.deleteHiveTables(StorageCleanupJob.java:409)
      at org.apache.kylin.rest.job.StorageCleanupJob.cleanUnusedIntermediateHiveTableInternal(StorageCleanupJob.java:375)
      at org.apache.kylin.rest.job.StorageCleanupJob.cleanUnusedIntermediateHiveTable(StorageCleanupJob.java:278)
      at org.apache.kylin.rest.job.StorageCleanupJob.cleanup(StorageCleanupJob.java:151)
      at org.apache.kylin.rest.job.StorageCleanupJob.execute(StorageCleanupJob.java:145)
      at org.apache.kylin.common.util.AbstractApplication.execute(AbstractApplication.java:37)
      at org.apache.kylin.tool.StorageCleanupJob.main(StorageCleanupJob.java:27)
      Caused by: java.io.IOException: error=7, Argument list too long
      at java.lang.UNIXProcess.forkAndExec(Native Method)
      at java.lang.UNIXProcess.<init>(UNIXProcess.java:247)
      at java.lang.ProcessImpl.start(ProcessImpl.java:134)
      at java.lang.ProcessBuilder.start(ProcessBuilder.java:1029)
      ... 10 moreĀ 

      Instead of composing one long command, storage cleanup need to generate a script and feed that into beeline or hive CLI.

      Attachments

        Issue Links

          Activity

            People

              codingforfun weibin0516
              seva_ostapenko Vsevolod Ostapenko
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: