Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-3996

Support multiple tables and scanners as input to the mapper in map/reduce jobs

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Closed
    • Critical
    • Resolution: Fixed
    • None
    • 0.94.5, 0.95.0
    • mapreduce
    • None
    • Reviewed
    • Hide
      Adds MultiTableInputFormat.

      Usage example:

      {code}
      Scan scan1 = new Scan();
      scan1.setStartRow(start1);
      scan1.setStopRow(end1);
      Scan scan2 = new Scan();
      scan2.setStartRow(start2);
      scan2.setStopRow(end2);
      MultiTableInputCollection mtic = new MultiTableInputCollection();
      mtic.Add(tableName1, scan1);
      mtic.Add(tableName2, scan2);
      TableMapReduceUtil.initTableMapperJob(mtic, TestTableMapper.class, Text.class, IntWritable.class, job1);
      {code}
      Show
      Adds MultiTableInputFormat. Usage example: {code} Scan scan1 = new Scan(); scan1.setStartRow(start1); scan1.setStopRow(end1); Scan scan2 = new Scan(); scan2.setStartRow(start2); scan2.setStopRow(end2); MultiTableInputCollection mtic = new MultiTableInputCollection(); mtic.Add(tableName1, scan1); mtic.Add(tableName2, scan2); TableMapReduceUtil.initTableMapperJob(mtic, TestTableMapper.class, Text.class, IntWritable.class, job1); {code}

    Description

      It seems that in many cases feeding data from multiple tables or multiple scanners on a single table can save a lot of time when running map/reduce jobs.
      I propose a new MultiTableInputFormat class that would allow doing this.

      Attachments

        1. 3996-v15.txt
          34 kB
          Lars Hofhansl
        2. 3996-0.94.txt
          33 kB
          Lars Hofhansl
        3. 3996-v14.txt
          33 kB
          Bryan Baugher
        4. 3996-v13.txt
          33 kB
          Bryan Baugher
        5. 3996-v12.txt
          33 kB
          Bryan Baugher
        6. 3996-v11.txt
          33 kB
          Bryan Baugher
        7. 3996-v10.txt
          33 kB
          Bryan Baugher
        8. 3996-v9.txt
          32 kB
          Bryan Baugher
        9. 3996-v8.txt
          32 kB
          Bryan Baugher
        10. 3996-v7.txt
          42 kB
          Ted Yu
        11. 3996-v6.txt
          40 kB
          Ted Yu
        12. 3996-v5.txt
          40 kB
          Eran Kutner
        13. 3996-v4.txt
          40 kB
          Ted Yu
        14. 3996-v3.txt
          42 kB
          Ted Yu
        15. 3996-v2.txt
          43 kB
          Ted Yu
        16. HBase-3996.patch
          66 kB
          Eran Kutner

        Issue Links

          is related to

          Bug - A problem which impairs or prevents the functions of the product. HIVE-4515 "select count(*) from table" query on hive-0.10.0, hbase-0.94.7 integration throws exceptions

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HIVE-4520 java.lang.NegativeArraySizeException when query on hive-0.11.0, hbase-0.94.6.1

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved

          Activity

            People

              bbaugher Bryan Baugher
              erank Eran Kutner
              Votes:
              10 Vote for this issue
              Watchers:
              13 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: