Details
-
Improvement
-
Status: Open
-
Major
-
Resolution: Unresolved
-
None
-
None
Description
Issue
As part of FLINK-31772
I performed a complete benchmark for KinesisStreamsSink after configuring rate limiting strategy.
It appears that optimum values for rate limiting strategy parameters are dependent on use case (shard number/ parallellism/ record thouroughput)
We initially implemeted the AIMDRateLimitingStrategy in accordance with one used for TCP congestion control but since parameters are use case dependent we would like to allow sink users to adjust parameters as suitable.
Requirements
- we must allow users to configure increment rate and decrease factor of AIMDRateLimitingStrategy for KinesisStreamsSink
- we must provide backward compatible default values identical to current values to introduce no further regressions.
Appendix
Performace Benchmark Results
| Parallelism/Shards/Payload | paralellism | shards | payload | records/sec | Async Sink | Async Sink With Configured Ratelimiting Strategy Thourouput (MB/s) | Async sink/ Maximum Thourouput | % of Improvement |
| Low/Low/Low | 1 | 1 | 1024 | 10000 | 0.991 | 1 | 1 | 0.9 |
| Low/Low/High | 1 | 1 | 102400 | 100 | 0.9943 | 1 | 1 | 0.57 |
| Low/Med/Low | 1 | 8 | 1024 | 80000 | 4.12 | 4.57 | 0.57125 | 5.625 |
| Low/Med/High | 1 | 8 | 102400 | 800 | 4.35 | 7.65 | 0.95625 | 41.25 |
| Med/Low/Low | 8 | 1 | 1024 | 20000 | 0.852 | 0.846 | 0.846 | -0.6 |
| Med/Low/High | 8 | 1 | 102400 | 200 | 0.921 | 0.867 | 0.867 | -5.4 |
| Med/Med/Low | 8 | 8 | 1024 | 80000 | 5.37 | 4.76 | 0.595 | -7.625 |
| Med/Med/High | 8 | 8 | 102400 | 800 | 7.53 | 7.69 | 0.96125 | 2 |
| Med/High/Low | 8 | 64 | 1024 | 80000 | 32.5 | 37.4 | 0.58438 | 7.65625 |
| Med/High/High | 8 | 64 | 102400 | 800 | 47.27 | 60.4 | 0.94375 | 20.51562 |
| High/High/Low | 256 | 256 | 1024 | 300000 | 127 | 127 | 0.49609 | 0 |
| High/High/High | 256 | 256 | 102400 | 3000 | 225 | 246 | 0.96094 | 8.20313 |