Uploaded image for project: 'Apache Tez'
  1. Apache Tez
  2. TEZ-1128

OnFileUnorderedPartitionedKVOutput does not handle partitioning correctly with the MRPartitioner

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 0.5.0
    • None
    • None

    Description

      Two issues:
      1) It does not set value of TezJobConfig.TEZ_RUNTIME_NUM_EXPECTED_PARTITIONS. So MRPartitioner assumes the number of partitions is 1 and goes with a default partitioner that writes only to last partition.
      2) If one of the partitions has empty output (due to issue 1) it gets stuck without fetching input.

      Attachments

        1. TEZ-1128.1.txt
          1 kB
          Siddharth Seth

        Issue Links

          blocks

          Sub-task - The sub-task of the issue PIG-3937 Use unsorted shuffle in Union

          • Major - Major loss of function.
          • Closed

          Activity

            People

              sseth Siddharth Seth
              rohini Rohini Palaniswamy
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: