Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-3211 Add AWS Kinesis streaming connector
  3. FLINK-4018

Configurable idle time between getRecords requests to Kinesis shards

    XMLWordPrintableJSON

Details

    Description

      Currently, the Kinesis consumer is calling getRecords() right after finishing previous calls. This results in easily reaching Amazon's limitation of 5 GET requests per shard per second. Although the code already has backoff & retry mechanism, this will affect other applications consuming from the same Kinesis stream.

      Along with this new configuration and the already existing `KinesisConfigConstants.CONFIG_SHARD_RECORDS_PER_GET`, users will have more control on the desired throughput behaviour for the Kinesis consumer.

      Attachments

        Issue Links

          Activity

            People

              tzulitai Tzu-Li (Gordon) Tai
              tzulitai Tzu-Li (Gordon) Tai
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: