Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-24438

Port Kinesis Source to new Source API (FLIP-27)

    XMLWordPrintableJSON

Details

    Description

      We want to port the Kinesis Source to the new Source API (FLIP-27) and deprecate the old FlinkKinesisConsumer

      Attachments

        Issue Links

          blocks

          Sub-task - The sub-task of the issue FLINK-32696 Promote Kinesis connector support to PublicEvolving

          • Major - Major loss of function.
          • Open
          duplicates

          Improvement - An improvement or enhancement to an existing feature or task. FLINK-25729 Replace the deprecated FlinkKinesisConsumer with a Kinesis Consumer based on the AWS SDK for Java 2.x

          • Major - Major loss of function.
          • Resolved
          is related to

          Bug - A problem which impairs or prevents the functions of the product. FLINK-36215 Child shards are getting started with LATEST shard iterator even when parents are not expired

          • Major - Major loss of function.
          • Resolved
          links to

          Web Link GitHub Pull Request #49

          1.
          		 Initial implementation of Kinesis Source using FLIP-27 Sub-task Resolved Hong Liang Teoh
          2.
          		 Port over Kinesis Client configurations for retry and backoff Sub-task Resolved Burak Ozakinci
          3.
          		 Implement support for EFO in new KDS Source Sub-task Open Hong Liang Teoh
          4.
          		 Implement Integration test for FLIP-27 DataStreams API Sub-task Open Elphas Toringepi
          5.
          		 Implement KDS Table API support Sub-task Open Burak Ozakinci
          6.
          		 Implement Python wrapper for new KDS source Sub-task Open Unassigned
          7.
          		 Update documentation Sub-task Open Unassigned
          8.
          		 Use Flink Configuration to specify KDS Source configuration object Sub-task Closed Hong Liang Teoh
          9.
          		 Implement support for Kinesis deaggregation Sub-task Open Unassigned
          10.
          		 Implement support for parent/child shard ordering Sub-task Resolved Aleksandr Pilipenko
          11.
          		 Implement metrics and logging for Initial implementation Sub-task Resolved Burak Ozakinci
          12.
          		 Implement watermark alignment on KDS and DDB source Sub-task Resolved Aleksandr Pilipenko
          13.
          		 Implement support for processing subset of shards/traffic Sub-task Open Unassigned
          14.
          		 Add connector abstraction layer to remove reliance on AWS SDK classes Sub-task Resolved Unassigned
          15.
          		 Implement DDB Streams Table API support Sub-task Open Unassigned
          16.
          		 Add support for DDB Streams for DataStream API Sub-task Resolved Abhi Gupta
          17.
          		 Address Shard Consistency Issue for DDB Streams Source Sub-task Resolved Abhishek Kumar
          18.
          		 Implement parent-child ordering for DDB Streams source Sub-task Resolved Abhi Gupta
          19.
          		 Use region from stream ARN to construct SDK client Sub-task Resolved Aleksandr Pilipenko
          20.
          		 Update to AWS SDKv2 2.26.19 Sub-task Resolved Burak Ozakinci
          21.
          		 Reimplement UniformShardAssigner using Kinesis shard hashes Sub-task Resolved Hong Liang Teoh
          22.
          		 Port SingleThreadMultiplexSourceReaderBase to new undeprecated interfaces Sub-task Resolved Hong Liang Teoh
          23.
          		 SDK retry for KDS connector Sub-task Resolved Unassigned

          Activity

            People

              liangtl Hong Liang Teoh
              martijnvisser Martijn Visser
              Votes:
              3 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated: