Uploaded image for project: 'Kylin'
  1. Kylin
  2. KYLIN-4520

Improve StorageCleanJobHbaseUtil to cleanup HBase tables in parallel

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • v3.1.0
    • Tools, Build and Test
    • None
    • Sprint 51

    Description

      The current StorageCleanJobHbaseUtil supports multi-threads, while from the execution logs, it still check one by one, so not really multi-threading. 

       

      Below is the logs that use 5 threads " -- thread 5":

       

      // code placeholder

2020-05-23 02:02:26,538 INFO  [main] job.StorageCleanJobHbaseUtil:114 : Use 5 threads to drop unused hbase tables
2020-05-23 02:02:26,539 INFO  [pool-5-thread-1] job.StorageCleanJobHbaseUtil:171 : Deleting HBase table KYLIN_56XU9WDLOZ
2020-05-23 02:02:26,562 INFO  [pool-5-thread-1] job.StorageCleanJobHbaseUtil:180 : HBase tableKYLIN_56XU9WDLOZ does not exist
2020-05-23 02:02:26,564 INFO  [pool-5-thread-2] job.StorageCleanJobHbaseUtil:171 : Deleting HBase table KYLIN_590IHJ5QTM
2020-05-23 02:02:26,576 INFO  [pool-5-thread-2] job.StorageCleanJobHbaseUtil:180 : HBase tableKYLIN_590IHJ5QTM does not exist
2020-05-23 02:02:26,577 INFO  [pool-5-thread-3] job.StorageCleanJobHbaseUtil:171 : Deleting HBase table KYLIN_7U8J7ZFBYN
2020-05-23 02:02:26,586 INFO  [pool-5-thread-3] job.StorageCleanJobHbaseUtil:180 : HBase tableKYLIN_7U8J7ZFBYN does not exist
2020-05-23 02:02:26,587 INFO  [pool-5-thread-4] job.StorageCleanJobHbaseUtil:171 : Deleting HBase table KYLIN_BOR03PP4PV
2020-05-23 02:02:26,595 INFO  [pool-5-thread-4] job.StorageCleanJobHbaseUtil:180 : HBase tableKYLIN_BOR03PP4PV does not exist
2020-05-23 02:02:26,596 INFO  [pool-5-thread-5] job.StorageCleanJobHbaseUtil:171 : Deleting HBase table KYLIN_ESGGE02OQI
2020-05-23 02:02:26,606 INFO  [pool-5-thread-5] job.StorageCleanJobHbaseUtil:180 : HBase tableKYLIN_ESGGE02OQI does not exist
2020-05-23 02:02:26,606 INFO  [pool-5-thread-1] job.StorageCleanJobHbaseUtil:171 : Deleting HBase table KYLIN_FHWPLEOSO6
2020-05-23 02:02:26,617 INFO  [pool-5-thread-1] job.StorageCleanJobHbaseUtil:180 : HBase tableKYLIN_FHWPLEOSO6 does not exist
2020-05-23 02:02:26,618 INFO  [pool-5-thread-2] job.StorageCleanJobHbaseUtil:171 : Deleting HBase table KYLIN_GWYKX70W9A
2020-05-23 02:02:26,642 INFO  [pool-5-thread-2] client.HBaseAdmin:1330 : Started disable of KYLIN_GWYKX70W9A
2020-05-23 02:02:30,802 INFO  [pool-5-thread-2] client.HBaseAdmin:1357 : Disabled KYLIN_GWYKX70W9A
2020-05-23 02:02:32,094 INFO  [pool-5-thread-2] client.HBaseAdmin:924 : Deleted KYLIN_GWYKX70W9A
2020-05-23 02:02:32,094 INFO  [pool-5-thread-2] job.StorageCleanJobHbaseUtil:178 : Deleted HBase table KYLIN_GWYKX70W9A
2020-05-23 02:02:32,095 INFO  [pool-5-thread-3] job.StorageCleanJobHbaseUtil:171 : Deleting HBase table KYLIN_HLB7508SUV
2020-05-23 02:02:32,113 INFO  [pool-5-thread-3] client.HBaseAdmin:1330 : Started disable of KYLIN_HLB7508SUV
2020-05-23 02:02:34,348 INFO  [pool-5-thread-3] client.HBaseAdmin:1357 : Disabled KYLIN_HLB7508SUV
2020-05-23 02:02:36,582 INFO  [pool-5-thread-3] client.HBaseAdmin:924 : Deleted KYLIN_HLB7508SUV
2020-05-23 02:02:36,582 INFO  [pool-5-thread-3] job.StorageCleanJobHbaseUtil:178 : Deleted HBase table KYLIN_HLB7508SUV
2020-05-23 02:02:36,582 INFO  [pool-5-thread-4] job.StorageCleanJobHbaseUtil:171 : Deleting HBase table KYLIN_JPIJ7GT1F5
2020-05-23 02:02:36,599 INFO  [pool-5-thread-4] client.HBaseAdmin:1330 : Started disable of KYLIN_JPIJ7GT1F5
2020-05-23 02:02:40,959 INFO  [pool-5-thread-4] client.HBaseAdmin:1357 : Disabled KYLIN_JPIJ7GT1F5
2020-05-23 02:02:42,119 INFO  [pool-5-thread-4] client.HBaseAdmin:924 : Deleted KYLIN_JPIJ7GT1F5
2020-05-23 02:02:42,119 INFO  [pool-5-thread-4] job.StorageCleanJobHbaseUtil:178 : Deleted HBase table KYLIN_JPIJ7GT1F5
2020-05-23 02:02:42,119 INFO  [pool-5-thread-5] job.StorageCleanJobHbaseUtil:171 : Deleting HBase table KYLIN_KV4U23OBEP
2020-05-23 02:02:42,141 INFO  [pool-5-thread-5] client.HBaseAdmin:1330 : Started disable of KYLIN_KV4U23OBEP
2020-05-23 02:02:46,322 INFO  [pool-5-thread-5] client.HBaseAdmin:1357 : Disabled KYLIN_KV4U23OBEP
2020-05-23 02:02:47,958 INFO  [pool-5-thread-5] client.HBaseAdmin:924 : Deleted KYLIN_KV4U23OBEP
2020-05-23 02:02:47,958 INFO  [pool-5-thread-5] job.StorageCleanJobHbaseUtil:178 : Deleted HBase table KYLIN_KV4U23OBEP
2020-05-23 02:02:47,958 INFO  [pool-5-thread-1] job.StorageCleanJobHbaseUtil:171 : Deleting HBase table KYLIN_L802CS1ZZU
2020-05-23 02:02:47,975 INFO  [pool-5-thread-1] client.HBaseAdmin:1330 : Started disable of KYLIN_L802CS1ZZU
2020-05-23 02:02:50,194 INFO  [pool-5-thread-1] client.HBaseAdmin:1357 : Disabled KYLIN_L802CS1ZZU
2020-05-23 02:02:51,554 INFO  [pool-5-thread-1] client.HBaseAdmin:924 : Deleted KYLIN_L802CS1ZZU
2020-05-23 02:02:51,555 INFO  [pool-5-thread-1] job.StorageCleanJobHbaseUtil:178 : Deleted HBase table KYLIN_L802CS1ZZU
2020-05-23 02:02:51,555 INFO  [pool-5-thread-2] job.StorageCleanJobHbaseUtil:171 : Deleting HBase table KYLIN_L9D9CZXZIG
2020-05-23 02:02:51,570 INFO  [pool-5-thread-2] client.HBaseAdmin:1330 : Started disable of KYLIN_L9D9CZXZIG
2020-05-23 02:02:53,773 INFO  [pool-5-thread-2] client.HBaseAdmin:1357 : Disabled KYLIN_L9D9CZXZIG

      Attachments

        Activity

          People

            shaofengshi Shao Feng Shi
            shaofengshi Shao Feng Shi
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: