Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 0.9.3, 0.10.2
    • None
    • None

    Description

      TestSecureShuffle is very slow. We need to experiment with it.
      There is already an optimization like:

          conf.setInt(TezRuntimeConfiguration.TEZ_RUNTIME_SHUFFLE_FETCH_FAILURES_LIMIT, 2);
      

      Maybe it is worth checking if lowering the number of failed attempts can help. In case of shuffle failures, sooner or later the reducer and mapper task is retried, so lowering:

      TezConfiguration.TEZ_AM_TASK_MAX_FAILED_ATTEMPTS
      

      might help, theoretically, as we want the DAG to fail as fast as possible.

      Attachments

        Issue Links

          Activity

            People

              mbathori Mark Bathori
              abstractdog László Bodor
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 40m
                  40m