Details
-
Sub-task
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
None
Description
Background
In order to add support for EFO in the FlinkKinesisConsumer we are abstracting out the record consumption from Kinesis ShardConsumer and introducing an interface.
Scope
Introduce the RecordPublisher interface and refactor the existing polling implementation to implement it:
- Add PollingRecordPublisher that is functionally equivalent to the existing implementation
- Support adaptive throughput via an extension, AdaptivePollingRecordSubscriber
- Split out the ShardMetricReporter into separate classes such that each component can report it's own metrics:
- ShardConsumer
- PollingRecordConsumer
- FanOutRecordConsumer (later)
- All the existing unit test will continue to pass, and be functionally equivalent (there may be minor compilation tweaks)
Attachments
Issue Links
- causes
-
FLINK-21661 SHARD_GETRECORDS_INTERVAL_MILLIS wrong use?
- Resolved
- links to