Uploaded image for project: 'Apache Tez'
  1. Apache Tez
  2. TEZ-4262

Integer overflow in unordered.output.buffer.size-mb

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Critical
    • Resolution: Duplicate
    • None
    • None
    • None
    • None

    Description

      The value of the tez.runtime.unordered.output.buffer.size-mb parameter exceeds the range of the int type。

      Exception thrown is given below:

      2020-12-30 14:57:37,762 [INFO] [Dispatcher thread \{Central}] |HistoryEventHandler.criticalEvents|: [HISTORY][DAG:dag_1609310178038_0003_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=Map 1, taskAttemptId=attempt_1609310178038_0003_1_00_000160_0, creationTime=1609311453532, allocationTime=1609311455291, startTime=1609311457280, finishTime=1609311457744, timeTaken=464, status=FAILED, taskFailureType=NON_FATAL, errorEnum=FRAMEWORK_ERROR, diagnostics=Error: Error while running task ( failure ) : attempt_1609310178038_0003_1_00_000160_0:java.lang.IllegalArgumentException2020-12-30 14:57:37,762 [INFO] [Dispatcher thread \{Central}] |HistoryEventHandler.criticalEvents|: [HISTORY][DAG:dag_1609310178038_0003_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=Map 1, taskAttemptId=attempt_1609310178038_0003_1_00_000160_0, creationTime=1609311453532, allocationTime=1609311455291, startTime=1609311457280, finishTime=1609311457744, timeTaken=464, status=FAILED, taskFailureType=NON_FATAL, errorEnum=FRAMEWORK_ERROR, diagnostics=Error: Error while running task ( failure ) : attempt_1609310178038_0003_1_00_000160_0:java.lang.IllegalArgumentException at com.google.common.base.Preconditions.checkArgument(Preconditions.java:108) at org.apache.tez.runtime.common.resources.MemoryDistributor.registerRequest(MemoryDistributor.java:177) at org.apache.tez.runtime.common.resources.MemoryDistributor.requestMemory(MemoryDistributor.java:110) at org.apache.tez.runtime.api.impl.TezTaskContextImpl.requestInitialMemory(TezTaskContextImpl.java:214) at org.apache.tez.runtime.library.output.UnorderedPartitionedKVOutput.initialize(UnorderedPartitionedKVOutput.java:76) at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask$InitializeOutputCallable._callInternal(LogicalIOProcessorRuntimeTask.java:550) at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask$InitializeOutputCallable.callInternal(LogicalIOProcessorRuntimeTask.java:533) at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask$InitializeOutputCallable.callInternal(LogicalIOProcessorRuntimeTask.java:518) at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)

      Attachments

        1. TEZ-4262.test.patch
          1 kB
          Jonathan Turner Eagles

        Issue Links

          Activity

            People

              it_singer yanbin.zhang
              it_singer yanbin.zhang
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 50m
                  50m