Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-2992 The transforms formerly known as non-lazy
  3. SPARK-2991

RDD transforms for scan and scanLeft

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Minor
    • Resolution: Won't Fix
    • None
    • None
    • Spark Core

    Description

      Provide RDD transforms analogous to Scala scan(z)(f) (parallel prefix scan) and scanLeft(z)(f) (sequential prefix scan)

      Discussion of a scanLeft implementation:
      http://erikerlandson.github.io/blog/2014/08/09/implementing-an-rdd-scanleft-transform-with-cascade-rdds/

      Discussion of scan:
      http://erikerlandson.github.io/blog/2014/08/12/implementing-parallel-prefix-scan-as-a-spark-rdd-transform/

      Attachments

        Issue Links

          relates to

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. SPARK-9999 Dataset API on top of Catalyst/DataFrame

          • Major - Major loss of function.
          • Resolved
          links to

          Pull request #1909 [Github] Pull Request #1909 (erikerlandson)

          Activity

            People

              eje Erik Erlandson
              eje Erik Erlandson
              Votes:
              2 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: