Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-35659

Avoid write null to StateStore

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.0.2, 3.1.2, 3.2.0
    • 3.0.3, 3.2.0, 3.1.3
    • Structured Streaming
    • None

    Description

      According to get method doc in StateStore API, it returns non-null row if the key exists. So basically we should avoid write null to StateStore. You cannot distinguish if the returned null row is because the key doesn't exist, or the value is actually null. And due to the defined behavior of get, it is quite easy to cause NPE error if the caller doesn't expect to get a null if the caller believes the key exists.

      Attachments

        Activity

          People

            viirya L. C. Hsieh
            viirya L. C. Hsieh
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: