[SPARK-39083] Fix FsHistoryProvider race condition between update and clean app data - ASF JIRA

XML

Word

Printable

JSON

Type: Bug
Status: Resolved
Priority: Major
Resolution: Fixed
Affects Version/s: 3.2.1
Fix Version/s: 3.3.0, 3.2.2
Component/s: Web UI
Labels:
None
Environment:

CentOS Linux release 7.7.1908

openjdk version "1.8.0_262"
OpenJDK Runtime Environment (build 1.8.0_262-b10)
OpenJDK 64-Bit Server VM (build 25.262-b10, mixed mode)

After ~~SPARK-29043~~, FsHistoryProvider will list the log info without waiting all `mergeApplicationListing` task finished.

However, the method `cleanAppData` for stale data is not thread safe and it could lead to a scenario when `cleanAppData` delete the entry of

ApplicationInfoWrapper for an application right after it has been updated by `mergeApplicationListing`.

So there will be cases when the HS Web UI displays `Application not found` for applications whose logs does exist.

is duplicated by

SPARK-39082 Fix FsHistoryProvider race condition between update and clean app data

relates to

SPARK-29043 [History Server]Only one replay thread of FsHistoryProvider work because of straggler

SPARK-37659 Fix FsHistoryProvider race condition between list and delet log info

links to

[Github] Pull Request #36424 (tanvn)