Details
Description
I compiled with the current branch-3.0 source and tested it in mac os. A java.lang.NullPointerException will be thrown when below conditions are met:
- Using InMemoryStore as kvstore when parsing the event log file (e.g., when spark.history.store.path is unset).
- At least one stage in this event log has task number greater than spark.ui.retainedTasks (by default is 100000). In this case, kvstore needs to delete extra task records.
- The job has more than one stage, so parentToChildrenMap in InMemoryStore.java will have more than one key.
The java.lang.NullPointerException is thrown in InMemoryStore.java :296. In the method deleteParentIndex().
private void deleteParentIndex(Object key) { if (hasNaturalParentIndex) { for (NaturalKeys v : parentToChildrenMap.values()) { if (v.remove(asKey(key))) { // `v` can be empty after removing the natural key and we can remove it from // `parentToChildrenMap`. However, `parentToChildrenMap` is a ConcurrentMap and such // checking and deleting can be slow. // This method is to delete one object with certain key, let's make it simple here. break; } } } }
In “if (v.remove(asKey(key)))”, if the key is not contained in v, "v.remove(asKey(key))" will return null, and java will throw a NullPointerException when executing "if (null)".
An exception stack trace is attached.
This issue can be fixed by updating if statement to
if (v.remove(asKey(key)) != null)
Attachments
Attachments
Issue Links
- links to