Uploaded image for project: 'Apache Tez'
  1. Apache Tez
  2. TEZ-1800

Integer overflow in ExternalSorter.getInitialMemoryRequirement()

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 0.5.4
    • None
    • None
    • Reviewed

    Description

      When trying to implement support for tez.runtime.io.sort.mb > 2 GB in pipelinedsorter, I encountered integer overflow exception in ExternalSorter.

      Exception thrown is given below.

      ], TaskAttempt 3 failed, info=[Error: Failure while running task:java.lang.IllegalArgumentException
              at com.google.common.base.Preconditions.checkArgument(Preconditions.java:72)
              at org.apache.tez.runtime.common.resources.MemoryDistributor.registerRequest(MemoryDistributor.java:158)
              at org.apache.tez.runtime.common.resources.MemoryDistributor.requestMemory(MemoryDistributor.java:97)
              at org.apache.tez.runtime.api.impl.TezTaskContextImpl.requestInitialMemory(TezTaskContextImpl.java:192)
              at org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput.initialize(OrderedPartitionedKVOutput.java:95)
              at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask$InitializeOutputCallable.call(LogicalIOProcessorRuntimeTask.java:430)
              at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask$InitializeOutputCallable.call(LogicalIOProcessorRuntimeTask.java:409)
              at java.util.concurrent.FutureTask.run(FutureTask.java:266)
              at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
              at java.util.concurrent.FutureTask.run(FutureTask.java:266)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)
      
      

      Actual place where the integer overflow happens is in ExternalSorter.getInitialMemoryRequirement()

      int initialMemRequestMb = 
              conf.getInt(
                  TezRuntimeConfiguration.TEZ_RUNTIME_IO_SORT_MB, 
                  TezRuntimeConfiguration.TEZ_RUNTIME_IO_SORT_MB_DEFAULT);
      long reqBytes = initialMemRequestMb << 20;
      

      Attachments

        1. TEZ-1800.1.patch
          0.9 kB
          Rajesh Balamohan

        Issue Links

          Activity

            sseth Siddharth Seth added a comment -

            +1.

            sseth Siddharth Seth added a comment - +1.

            Thanks sseth. Committed to master.
            >>>
            commit 79edb49b245bb274ddb377d53683187f8a640a0d
            >>>

            rajesh.balamohan Rajesh Balamohan added a comment - Thanks sseth . Committed to master. >>> commit 79edb49b245bb274ddb377d53683187f8a640a0d >>>
            hitesh Hitesh Shah added a comment -

            This should be backported to 0.5.x.

            hitesh Hitesh Shah added a comment - This should be backported to 0.5.x.

            committed to branch-0.5

            >>>
            commit 1fb43bd29e20b1718e6652c5175ffe9b2d05fac1
            >>>

            rajesh.balamohan Rajesh Balamohan added a comment - committed to branch-0.5 >>> commit 1fb43bd29e20b1718e6652c5175ffe9b2d05fac1 >>>
            sseth Siddharth Seth added a comment -

            Setting fix version to 0.5.4 since this is already in branch-0.5

            sseth Siddharth Seth added a comment - Setting fix version to 0.5.4 since this is already in branch-0.5
            hitesh Hitesh Shah added a comment -

            Closing issue as 0.5.4, 0.6.1 and 0.7.0 have been released.

            hitesh Hitesh Shah added a comment - Closing issue as 0.5.4, 0.6.1 and 0.7.0 have been released.

            People

              rajesh.balamohan Rajesh Balamohan
              rajesh.balamohan Rajesh Balamohan
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: