Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-74

Hive can use CombineFileInputFormat for when the input are many small files

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 0.5.0
    • Query Processor
    • None
    • Reviewed
    • Hide
      HIVE-74. Hive can use CombineFileInputFormat for when the input has many
      small files (Namit Jain via rmurthy)
      Show
      HIVE-74 . Hive can use CombineFileInputFormat for when the input has many small files (Namit Jain via rmurthy)

    Description

      There are cases when the input to a Hive job are thousands of small files. In this case, there is a mapper for each file. Most of the overhead for spawning all these mappers can be avoided if Hive used CombineFileInputFormat introduced via HADOOP-4565

      Options to control this behavior:

      hive.input.format (org.apache.hadoop.hive.ql.io.CombineHiveInputFormat (default, if empty), or org.apache.hadoop.hive.ql.io.HiveInputFormat)
mapred.min.split.size.per.node (the minimum bytes of data to create a node-local partition, otherwise the data will combine to rack level. default:0)
mapred.min.split.size.per.rack (the minimum bytes of data to create a rack-local partition, otherwise the data will combine to global level. default:0)
mapred.max.split.size (the max size of each split, will be exceeded because we stop accumulating *after* reaching it, instead of before)

      The 3 numbers above must be in non-descending order.

      Attachments

        1. hiveCombineSplit2.patch
          15 kB
          Dhruba Borthakur
        2. hiveCombineSplit.patch
          14 kB
          Dhruba Borthakur
        3. hiveCombineSplit.patch
          15 kB
          Dhruba Borthakur
        4. hive.74.2.patch
          37 kB
          Namit Jain
        5. hive.74.1.patch
          36 kB
          Namit Jain

        Issue Links

          blocks

          Bug - A problem which impairs or prevents the functions of the product. HIVE-826 cleanup HiveInputFormat.getRecordReader()

          • Major - Major loss of function.
          • Open
          is blocked by

          Improvement - An improvement or enhancement to an existing feature or task. HADOOP-4565 MultiFileInputSplit can use data locality information to create splits

          • Major - Major loss of function.
          • Closed
          is related to

          Bug - A problem which impairs or prevents the functions of the product. HIVE-826 cleanup HiveInputFormat.getRecordReader()

          • Major - Major loss of function.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. HIVE-824 use same mapper for mltiple directories

          • Major - Major loss of function.
          • Open
          relates to

          Improvement - An improvement or enhancement to an existing feature or task. HIVE-824 use same mapper for mltiple directories

          • Major - Major loss of function.
          • Open

          Activity

            People

              rajesh.balamohan Rajesh Balamohan
              dhruba Dhruba Borthakur
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: