Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-7398 Add back-pressure to Spark Streaming (umbrella JIRA)
  3. SPARK-8835

Provide pluggable Congestion Strategies to deal with Streaming load

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Incomplete
    • None
    • None
    • DStreams

    Description

      Second part of SPARK-7398 (which has an over-arching, high-level design doc).

      An (outdated) PoC implementation exists.

      Note that throttling in the current (Spark 1.5.0) way pushes down the (fast-overflowing) buffer of received elements down to the TCP stack, which then has to deal with the break. This can be witnessed here :

      https://github.com/skyluc/spark-streaming-testbed/tree/test-runs-006/test-runs-006

      Attachments

        Issue Links

          is blocked by

          Sub-task - The sub-task of the issue SPARK-8978 Implement the DirectKafkaRateController

          • Major - Major loss of function.
          • Resolved

          Sub-task - The sub-task of the issue SPARK-8979 Implement a PIDRateEstimator

          • Major - Major loss of function.
          • Resolved

          New Feature - A new feature of the product, which has yet to be developed. SPARK-8834 Throttle DStreams dynamically through back-pressure information

          • Major - Major loss of function.
          • Closed
          links to

          Pull request #9200 [Github] Pull Request #9200 (huitseeker)

          Pull request #20478 [Github] Pull Request #20478 (EmergentOrder)

          Web Link GitHub Pull Request #20478

          Activity

            People

              Unassigned Unassigned
              huitseeker François Garillot
              Tathagata Das Tathagata Das
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: