Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-14423

S3: Fetching partition sizes from FS can be expensive when stats are not available in metastore

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Closed
    • Minor
    • Resolution: Fixed
    • None
    • 2.2.0
    • None

    Description

      When partition stats are not available in metastore, it tries to get the file sizes from FS.

      e.g

              at org.apache.hadoop.fs.FileSystem.getContentSummary(FileSystem.java:1487)
        at org.apache.hadoop.hive.ql.stats.StatsUtils.getFileSizeForPartitions(StatsUtils.java:598)
        at org.apache.hadoop.hive.ql.stats.StatsUtils.collectStatistics(StatsUtils.java:235)
        at org.apache.hadoop.hive.ql.stats.StatsUtils.collectStatistics(StatsUtils.java:144)
        at org.apache.hadoop.hive.ql.stats.StatsUtils.collectStatistics(StatsUtils.java:132)
        at org.apache.hadoop.hive.ql.optimizer.stats.annotation.StatsRulesProcFactory$TableScanStatsRule.process(StatsRulesProcFactory.java:126)
        at org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90)
        at org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:105)

      This can be quite expensive in some FS like S3. Especially when table is partitioned (e.g TPC-DS store_sales which has 1000s of partitions), query can spend 1000s of seconds just waiting for these information to be pulled in.

      Also, it would be good to remove FS.getContentSummary usage to find out file sizes.

      Attachments

        1. HIVE-14423.1.patch
          5 kB
          Rajesh Balamohan
        2. HIVE-14423.2.patch
          4 kB
          Rajesh Balamohan

        Issue Links

          is depended upon by

          Sub-task - The sub-task of the issue HADOOP-13525 Optimize uses of FS operations in the ASF analysis frameworks and libraries

          • Major - Major loss of function.
          • Resolved
          is related to

          Improvement - An improvement or enhancement to an existing feature or task. HIVE-13925 ETL optimizations

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Open
          relates to

          Sub-task - The sub-task of the issue HIVE-13901 Hivemetastore add partitions can be slow depending on filesystems

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Closed

          Activity

            People

              rajesh.balamohan Rajesh Balamohan
              rajesh.balamohan Rajesh Balamohan
              Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: