Details
-
Bug
-
Status: Resolved
-
P1
-
Resolution: Fixed
-
None
-
None
Description
In processElement:
private static class ReadFn<T> extends DoFn<Read<T>, T> { @ProcessElement public void processElement(@Element Read<T> spec, OutputReceiver<T> out) { AmazonDynamoDB client = spec.getAwsClientsProvider().createDynamoDB(); ScanRequest scanRequest = spec.getScanRequestFn().apply(null); scanRequest.setSegment(spec.getSegmentId()); ScanResult scanResult = client.scan(scanRequest); out.output(spec.getScanResultMapperFn().apply(scanResult)); } }
if total requested items > 1M dynamodb will only send upto 1M of data and expects client to read remaining items through LastEvaluatedKey (see this)
Attachments
Issue Links
- links to