Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-16264

Allow the user to use operators on the received DataFrame

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Closed
    • Major
    • Resolution: Won't Fix
    • None
    • None
    • SQL
    • None

    Description

      Currently Sink cannot apply any operators on the given DataFrame because new DataFrame created by the operator will use QueryExecution rather than IncrementalExecution.

      There are two options to fix this one:

      1. Merge IncrementalExecution into QueryExecution so that QueryExecution can also deal with streaming operators.
      2. Make Dataset operators inherits the QueryExecution(IncrementalExecution is just a subclass of IncrementalExecution) from it's parent.

      Attachments

        Issue Links

          is related to

          Sub-task - The sub-task of the issue SPARK-16020 Fix complete mode aggregation with console sink

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. SPARK-16350 Complete output mode does not output updated aggregated value in Structured Streaming

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. SPARK-16545 Structured Streaming : foreachSink creates the Physical Plan multiple times per TriggerInterval

          • Major - Major loss of function.
          • Resolved

          Activity

            People

              Unassigned Unassigned
              zsxwing Shixiong Zhu
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: