JDK-8194653: Deadlock involving FileSystems.getDefault and System.loadLibrary call

Instruction

This will cause deadlock and hangs concurrent tasks forever on the same executor. for example,

In the Spark UI stage tab, you may find some of the tasks hang for hours and all others complete without delay.

Also, you may find that these hanging tasks belong to the same executors.
Usually, in this case, you will also get nothing helpful from the executor log.

If you print the executor jstack or you check the ThreadDump via SparkUI executor tab and you find some task thread blocked like below, you are very likely to hit the JDK-8194653 issue.

Solutions

Here are some options to circumvent this problem:

1. For the cluster managers side, you can update the JDK version according to https://bugs.openjdk.java.net/browse/JDK-8194653
2. If you are not able to update the JDK version for the cluster entirely, you can use `spark.executorEnv.JAVA_HOME` to specify a suitable JRE for your apps
2. Also, turn on `spark.speculation` may let spark automatically re-run the hanging tasks and bypass the problem