Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-34023

Expose Kinesis client retry config in sink

    XMLWordPrintableJSON

Details

    Description

      The consumer side exposes client retry configuration like flink.shard.getrecords.maxretries but the producer side lacks similar config for PutRecords.

      The KinesisStreamsSinkWriter constructor calls

      this.httpClient = AWSGeneralUtil.createAsyncHttpClient(kinesisClientProperties);
      this.kinesisClient = buildClient(kinesisClientProperties, this.httpClient);
      

      But those methods only refer to these values (aside from endpoint/region/creds) in the kinesisClientProperties:

      • aws.http-client.max-concurrency
      • aws.http-client.read-timeout
      • aws.trust.all.certificates
      • aws.http.protocol.version

      Without control over retry, users can observe exceptions like

      Request attempt 2 failure: Unable to execute HTTP request: connection timed out after 2000 ms: kinesis.us-west-2.amazonaws.com

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              atchesonaws Brad Atcheson
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated: