[SPARK-46861] Avoid Deadlock in DAGScheduler - ASF JIRA

XML

Word

Printable

JSON

The DAGScheduler could currently run into a deadlock with another thread if both access the partitions of the same RDD at the same time.
To make progress in getCacheLocs, we require both exclusive access to the RDD partitions and the location cache. We first lock on the location cache, and then on the RDD.
When accessing partitions of an RDD, the RDD first acquires exclusive access on the partitions, and then might acquire exclusive access on the location cache.
If thread 1 is able to acquire access on the RDD, while thread 2 holds the access to the location cache, we can run into a deadlock situation.

links to

GitHub Pull Request #44882