Details
-
Bug
-
Status: Resolved
-
P2
-
Resolution: Fixed
-
None
-
None
Description
KafkaIO has a validation check to allow only certain runners capable of provide exactly-once semantics:
if ("org.apache.beam.runners.direct.DirectRunner".equals(runner) || runner.startsWith("org.apache.beam.runners.dataflow.") || runner.startsWith("org.apache.beam.runners.spark.") { ...
The Flink supports exactly-once checkpointing but the Flink Runner can't utilize it in the way KafkaIO intends it.
I think we should remove the check in favor of checking for translation of @RequiresStableInput. Changes to KafkaIO might have to be made to support EOS efficiently with the Flink Runner.
Attachments
Issue Links
- is related to
-
BEAM-5359 Support @RequiresStableInput on Flink runner
- Resolved
- links to