Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-3626

Replace AsyncRDDActions with a more general async. API

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Won't Fix
    • None
    • None
    • Spark Core
    • None

    Description

      The experimental AsyncRDDActions APIs seem to only exist in order to enable job cancellation.

      We've been considering extending these APIs to support progress monitoring, but this would require stabilizing them so they're no longer @Experimental.

      Instead, I propose to replace all of the AsyncRDDActions with a mechanism based on job groups which allows arbitrary computations to be run in job groups and supports cancellation / monitoring of Spark jobs launched from those computations.

      (full design pending; see my GitHub PR for more details).

      Attachments

        Issue Links

          Activity

            People

              joshrosen Josh Rosen
              joshrosen Josh Rosen
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: