Uploaded image for project: 'Apache Tez'
  1. Apache Tez
  2. TEZ-1800

Integer overflow in ExternalSorter.getInitialMemoryRequirement()

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 0.5.4
    • None
    • None
    • Reviewed

    Description

      When trying to implement support for tez.runtime.io.sort.mb > 2 GB in pipelinedsorter, I encountered integer overflow exception in ExternalSorter.

      Exception thrown is given below.

      ], TaskAttempt 3 failed, info=[Error: Failure while running task:java.lang.IllegalArgumentException
              at com.google.common.base.Preconditions.checkArgument(Preconditions.java:72)
              at org.apache.tez.runtime.common.resources.MemoryDistributor.registerRequest(MemoryDistributor.java:158)
              at org.apache.tez.runtime.common.resources.MemoryDistributor.requestMemory(MemoryDistributor.java:97)
              at org.apache.tez.runtime.api.impl.TezTaskContextImpl.requestInitialMemory(TezTaskContextImpl.java:192)
              at org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput.initialize(OrderedPartitionedKVOutput.java:95)
              at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask$InitializeOutputCallable.call(LogicalIOProcessorRuntimeTask.java:430)
              at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask$InitializeOutputCallable.call(LogicalIOProcessorRuntimeTask.java:409)
              at java.util.concurrent.FutureTask.run(FutureTask.java:266)
              at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
              at java.util.concurrent.FutureTask.run(FutureTask.java:266)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)
      
      

      Actual place where the integer overflow happens is in ExternalSorter.getInitialMemoryRequirement()

      int initialMemRequestMb = 
              conf.getInt(
                  TezRuntimeConfiguration.TEZ_RUNTIME_IO_SORT_MB, 
                  TezRuntimeConfiguration.TEZ_RUNTIME_IO_SORT_MB_DEFAULT);
      long reqBytes = initialMemRequestMb << 20;
      

      Attachments

        1. TEZ-1800.1.patch
          0.9 kB
          Rajesh Balamohan

        Issue Links

          Activity

            People

              rajesh.balamohan Rajesh Balamohan
              rajesh.balamohan Rajesh Balamohan
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: