Details
-
Improvement
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
2.3.0
-
None
Description
In the shuffle layer, we guarantee that zero-size blocks will never be requested (a block containing zero records is always 0 bytes in size and is marked as empty such that it will never be legitimately requested by executors). However, we failed to take advantage of this in the shuffle-read path: the existing code did not explicitly check whether blocks are non-zero-size.
We should add `buf.size != 0` checks to ShuffleBlockFetcherIterator to take advantage of this invariant and prevent potential data loss / corruption issues.
Attachments
Issue Links
- is related to
-
SPARK-27216 Upgrade RoaringBitmap to 0.7.45 to fix Kryo unsafe ser/dser issue
- Resolved
-
SPARK-27530 FetchFailedException: Received a zero-size buffer for block shuffle
- Resolved
- links to