Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-2259

Reshuffle may set watermark holds past the end of time

Details

    • Bug
    • Status: Resolved
    • P2
    • Resolution: Fixed
    • None
    • 2.0.0
    • sdk-java-core

    Description

      When it does so, the ReduceFnRunner throws an exception that the watermark hold is past the end of time

      This affects all non-Dataflow runners.

      (With additional checks)
      Caused by: java.lang.IllegalArgumentException: Assigned TimestampedValueInSingleWindow{value=KV

      {key: gameexamples:test_tgroh.l eaderboard_team shard: 33, org.apache.beam.sdk.io.gcp.bigquery.TableRowInfo@31436a52}

      , timestamp=2017-05-11T15:29:59.999Z, window=[20
      17-05-11T15:15:00.000Z..2017-05-11T15:30:00.000Z), pane=PaneInfo{isFirst=true, timing=EARLY, index=0}} to a window [2017-05-11T15:15:
      00.000Z..2017-05-11T15:30:00.000Z) with fn org.apache.beam.sdk.util.IdentityWindowFn@58543f9f. Expiry is past end of time. Windowing
      strategy WindowingStrategy

      {windowFn=org.apache.beam.sdk.util.IdentityWindowFn@58543f9f, allowedLateness=PT9223372036854.775S, trigger =ReshuffleTrigger(), accumulationMode=DISCARDING_FIRED_PANES, timestampCombiner=EARLIEST}

      Attachments

        Activity

          People

            tgroh Thomas Groh
            tgroh Thomas Groh
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: