Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-8834

Throttle DStreams dynamically through back-pressure information

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Closed
    • Major
    • Resolution: Duplicate
    • None
    • None
    • DStreams
    • None

    Description

      This aims to have Spark Streaming be more resilient to high-throughput situations through back-pressure signaling & dynamic throttling.

      The Design doc can be found there:
      https://issues.apache.org/jira/browse/SPARK-8834
      An (outdated) PoC implementation exists.

      Attachments

        Issue Links

          blocks

          Sub-task - The sub-task of the issue SPARK-8835 Provide pluggable Congestion Strategies to deal with Streaming load

          • Major - Major loss of function.
          • Resolved
          is superceded by

          Sub-task - The sub-task of the issue SPARK-8975 Implement a mechanism to send a new rate from the driver to the block generator

          • Major - Major loss of function.
          • Resolved

          Sub-task - The sub-task of the issue SPARK-8977 Define the RateEstimator interface, and implement the ReceiverRateController

          • Major - Major loss of function.
          • Resolved

          Sub-task - The sub-task of the issue SPARK-8978 Implement the DirectKafkaRateController

          • Major - Major loss of function.
          • Resolved

          Sub-task - The sub-task of the issue SPARK-8979 Implement a PIDRateEstimator

          • Major - Major loss of function.
          • Resolved
          supercedes

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-6691 Abstract and add a dynamic RateLimiter for Spark Streaming

          • Major - Major loss of function.
          • Resolved

          Activity

            People

              Unassigned Unassigned
              huitseeker François Garillot
              Tathagata Das Tathagata Das
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: