Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-9897

Further enhance adaptive reads in Kinesis Connector to depend on run loop time

    XMLWordPrintableJSON

Details

    Description

      In FLINK-9692, we introduced the ability for the shardConsumer to adaptively read more records based on the current average record size to optimize the 2 Mb/sec shard limit. The feature maximizes  maxNumberOfRecordsPerFetch of 5 reads/sec (as prescribed by Kinesis limits). In the case where applications take more time to process records in the run loop, they are no longer able to read at a frequency of 5 reads/sec (even though their fetchIntervalMillis maybe set to 200 ms). In such a scenario, the maxNumberOfRecordsPerFetch should be calculated based on the time that the run loop actually takes as opposed to fetchIntervalMillis. 

      Attachments

        Activity

          People

            glaksh100 Lakshmi Rao
            glaksh100 Lakshmi Rao
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: