Details

    Description

      Background

      In order to add support for EFO in the FlinkKinesisConsumer we are abstracting out the record consumption from Kinesis ShardConsumer and introducing an interface. 

      Scope

      Introduce the RecordPublisher interface and refactor the existing polling implementation to implement it:

      • Add PollingRecordPublisher that is functionally equivalent to the existing implementation
      • Support adaptive throughput via an extension, AdaptivePollingRecordSubscriber
      • Split out the ShardMetricReporter into separate classes such that each component can report it's own metrics:
        • ShardConsumer
        • PollingRecordConsumer
        • FanOutRecordConsumer (later)
      • All the existing unit test will continue to pass, and be functionally equivalent (there may be minor compilation tweaks)

      Attachments

        Issue Links

          Activity

            People

              danny.cranmer Danny Cranmer
              danny.cranmer Danny Cranmer
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: