Details
-
Bug
-
Status: Resolved
-
P2
-
Resolution: Fixed
-
2.23.0
-
None
Description
Internally, DynamoDBIO.Write uses the batchWriteItem method from the AWS SDK to sink items. However, there is a limitation in the AWS SDK that a call to batchWriteItem cannot contain duplicate keys.
Currently DynamoDBIO.Write performs no key deduplication before flushing a batch, which could cause ValidationException: Provided list of item keys contains duplicates, if consecutive updates to a single key is within the batch size (currently hardcoded to be 25).
To fix this bug, the batch of write requests need to be deduplicated before being sent to batchRequest.addRequestItemsEntry
Attachments
Issue Links
- relates to
-
BEAM-13009 DynamoDBIO misses writing items if `withDeduplicateKeys` is not set
- Triage Needed
- links to