Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-1052

UnboundedSource splitId uniqueness breaks if more than one source is used.

Details

    • Bug
    • Status: Resolved
    • P2
    • Resolution: Fixed
    • None
    • 0.4.0
    • runner-spark
    • None

    Description

      We use a "running-id" to identify source splits, but we reiterate for each source evaluated.
      Spark already assigns a unique id per InputDStream, it would be unique enough if we replace MicrobatchSource hash code with one containing both the running-id and the InputDStream id.

      Attachments

        Issue Links

          Activity

            People

              aviemzur Aviem Zur
              amitsela Amit Sela
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: