Uploaded image for project: 'Apache Tez'
  1. Apache Tez
  2. TEZ-3126

Log reason for not reducing parallelism

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Minor
    • Resolution: Fixed
    • None
    • 0.7.1, 0.8.3
    • None
    • None

    Description

      For example, when reducing parallelism from 36 to 22. The basePartitionRange will be 1 and will not re-configure the vertex.

      ShuffleVertexManager#determineParallelismAndApply
          int desiredTaskParallelism = 
              (int)(
                  (expectedTotalSourceTasksOutputSize+desiredTaskInputDataSize-1)/
                  desiredTaskInputDataSize);
          if(desiredTaskParallelism < minTaskParallelism) {
            desiredTaskParallelism = minTaskParallelism;
          }
          
          if(desiredTaskParallelism >= currentParallelism) {
            return true;
          }
          
          // most shufflers will be assigned this range
          basePartitionRange = currentParallelism/desiredTaskParallelism;
          
          if (basePartitionRange <= 1) {
            // nothing to do if range is equal 1 partition. shuffler does it by default
            return true;
          }
      

      Attachments

        1. TEZ-3126.2.patch
          3 kB
          Jonathan Turner Eagles
        2. TEZ-3126.1.patch
          1.0 kB
          Jonathan Turner Eagles

        Issue Links

          Activity

            People

              jeagles Jonathan Turner Eagles
              jeagles Jonathan Turner Eagles
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: