Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-5465

Tasks are often killed before they exit on their own

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 2.8.0, 3.0.0-alpha1
    • mr-am, mrv2
    • None
    • Reviewed

    Description

      If there is profiling enabled for mapper or reducer then hprof dumps profile.out at process exit. It is dumped after task signaled to AM that work is finished.

      AM kills container with finished work without waiting for hprof to finish dumps. If hprof is dumping larger outputs (such as with depth=4 while depth=3 works) , it could not finish dump in time before being killed making entire dump unusable because cpu and heap stats are missing.

      There needs to be better delay before container is killed if profiling is enabled.

      Attachments

        1. MAPREDUCE-5465.patch
          3 kB
          Ravi Prakash
        2. MAPREDUCE-5465-2.patch
          36 kB
          Ming Ma
        3. MAPREDUCE-5465-3.patch
          41 kB
          Ming Ma
        4. MAPREDUCE-5465-4.patch
          89 kB
          Ming Ma
        5. MAPREDUCE-5465-5.patch
          90 kB
          Ming Ma
        6. MAPREDUCE-5465-6.patch
          90 kB
          Ming Ma
        7. MAPREDUCE-5465-7.patch
          86 kB
          Ming Ma
        8. MAPREDUCE-5465-8.patch
          87 kB
          Ming Ma
        9. MAPREDUCE-5465-9.patch
          87 kB
          Ming Ma
        10. MAPREDUCE-5465-branch-2.patch
          86 kB
          Ming Ma

        Issue Links

          Activity

            People

              mingma Ming Ma
              hsn Radim Kolar
              Votes:
              0 Vote for this issue
              Watchers:
              15 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: