Details
-
Sub-task
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
2.3.0
-
None
Description
Any network request which does not use stream-to-disk that is sending over 2GB is doomed to fail, so we might as well at least set the default value of spark.maxRemoteBlockSizeFetchToMem to something < 2GB.
It probably makes sense to set it to something even lower still, but that might require more careful testing; this is a totally safe first step.