Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-3211 Add AWS Kinesis streaming connector
  3. FLINK-3229

Kinesis streaming consumer with integration of Flink's checkpointing mechanics

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 1.0.0
    • 1.1.0
    • Connectors / Common
    • None

    Description

      Opening a sub-task to implement data source consumer for Kinesis streaming connector (https://issues.apache.org/jira/browser/FLINK-3211).

      An example of the planned user API for Flink Kinesis Consumer:

      Properties kinesisConfig = new Properties();
config.put(KinesisConfigConstants.CONFIG_AWS_REGION, "us-east-1");
config.put(KinesisConfigConstants.CONFIG_AWS_CREDENTIALS_PROVIDER_TYPE, "BASIC");
config.put(
    KinesisConfigConstants.CONFIG_AWS_CREDENTIALS_PROVIDER_BASIC_ACCESSKEYID, 
    "aws_access_key_id_here");
config.put(
    KinesisConfigConstants.CONFIG_AWS_CREDENTIALS_PROVIDER_BASIC_SECRETKEY,
    "aws_secret_key_here");
config.put(KinesisConfigConstants.CONFIG_STREAM_INIT_POSITION_TYPE, "LATEST"); // or TRIM_HORIZON

DataStream<T> kinesisRecords = env.addSource(new FlinkKinesisConsumer<>(
    "kinesis_stream_name",
    new SimpleStringSchema(),
    kinesisConfig));

      Attachments

        Issue Links

          links to

          Web Link GitHub Pull Request #1911

          Activity

            People

              tzulitai Tzu-Li (Gordon) Tai
              tzulitai Tzu-Li (Gordon) Tai
              Votes:
              1 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: