Details
-
New Feature
-
Status: Closed
-
Major
-
Resolution: Fixed
-
None
Description
Motivation
User stories:
As a Flink user, I’d like to use Kinesis Firehose as sink for my data pipeline.
Scope:
- Implement an asynchronous sink for Kinesis Firehose by inheriting the AsyncSinkBase class. The implementation can for now reside in its own module in flink-connectors. The module and package name can be anything reasonable e.g. flink-connector-aws-kinesis for the module name and org.apache.flink.connector.aws.kinesis for the package name.
- The implementation must use the Kinesis Java Client.
- The implementation must allow users to configure the Kinesis Client, with reasonable default settings.
- Implement an asynchornous sink writer for Firehose by extending the AsyncSinkWriter. The implementation must deal with failed requests and retry them using the requeueFailedRequestEntry method. If possible, the implementation should batch multiple requests (PutRecordsRequestEntry objects) to Firehose for increased throughput. The implemented Sink Writer will be used by the Sink class that will be created as part of this story.
- Unit/Integration testing. Use Kinesalite (in-memory Kinesis simulation). We already use this in KinesisTableApiITCase.
- Java / code-level docs.
- End to end testing: add tests that hits a real AWS instance. (How to best donate resources to the Flink project to allow this to happen?)
References
More details to be found https://cwiki.apache.org/confluence/display/FLINK/FLIP-171%3A+Async+Sink
Attachments
Issue Links
- blocks
-
FLINK-25610 [FLIP-171] Kinesis Firehose implementation of Async Sink Table API
- Resolved
- Dependent
-
FLINK-24041 [FLIP-171] Generic AsyncSinkBase
- Resolved
- is depended upon by
-
FLINK-25692 Documentation for Kinesis Firehose Sink
- Closed
- relates to
-
FLINK-25943 Kinesis, Firehose Async Sink state serializer implementation
- Resolved
-
FLINK-25924 KDF Integration tests intermittently fails
- Closed
- supercedes
-
FLINK-8314 Add support for Kinesis Firehose
- Resolved
- links to
- mentioned in
-
Page Loading...