Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-34198

Add RocksDB StateStore implementation

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.2.0
    • None
    • Structured Streaming
    • None

    Description

      Currently Spark SS only has one built-in StateStore implementation HDFSBackedStateStore. Actually it uses in-memory map to store state rows. As there are more and more streaming applications, some of them requires to use large state in stateful operations such as streaming aggregation and join.

      Several other major streaming frameworks already use RocksDB for state management. So it is proven to be good choice for large state usage. But Spark SS still lacks of a built-in state store for the requirement.

      We would like to explore the possibility to add RocksDB-based StateStore into Spark SS.

       

      Attachments

        Issue Links

          Activity

            People

              XuanYuan Yuanjian Li
              viirya L. C. Hsieh
              L. C. Hsieh L. C. Hsieh
              Votes:
              2 Vote for this issue
              Watchers:
              13 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: