Uploaded image for project: 'Apache Nemo'
  1. Apache Nemo
  2. NEMO-444

Implement spill mechanism on Nemo

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Open
    • Major
    • Resolution: Unresolved
    • None
    • None

    Description

      Currently, Nemo doesn't have a spill mechanism. This makes executors prone to memory problems such as OOM(Out Of Memory) or GC when task data is large. For example, handling skewed shuffle data in Nemo results in OOM and executor failure, as all data has to be handled in-memory.

      We need to spill in-memory data to secondary storage when there are not enough memory in executor.

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              jeongyoon Jeongyoon Eo
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated: