Uploaded image for project: 'Apache Nemo'
  1. Apache Nemo
  2. NEMO-466

Dynamic Work Stealing on Nemo for handling skews

    XMLWordPrintableJSON

Details

    • Umbrella
    • Status: Open
    • Major
    • Resolution: Unresolved
    • None
    • None

    Description

      We aim to handle the problem on throttled resources (heterogeneous resources) and skewed input data. In order to solve this problem, we suggest dynamic work stealing that can dynamically track task statuses and steal workloads among each other. To do this, we have the following action items:

      • Dynamically collecting task statistics during execution
      • Detecting skewed tasks periodically
      • Splitting the data allocated in skewed tasks and reallocating them into new tasks
      • Synchronizing the optimization procedure
      • Evaluation of the resulting implementations

      Attachments

        Issue Links

          requires

          New Feature - A new feature of the product, which has yet to be developed. NEMO-480 Dynamic work stealing for streaming workloads

          • Major - Major loss of function.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. NEMO-467 Collect task statistics necessary for estimating duration

          • Major - Major loss of function.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. NEMO-468 Detect skewed task periodically

          • Major - Major loss of function.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. NEMO-469 Enable splitting tasks in execution

          • Major - Major loss of function.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. NEMO-470 Synchronize work stealing process between task executors

          • Major - Major loss of function.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. NEMO-471 Evaluate the performance of Work Stealing implementation

          • Major - Major loss of function.
          • Open

          Activity

            People

              Unassigned Unassigned
              wonook Wonook
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated: