Details
-
Improvement
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
3.0.0
Description
At all current Spark releases when event logging on spark streaming is enabled the event logs grow massively. The files continue to grow until the application is stopped or killed.
The Spark history server then has difficulty processing the files.
https://issues.apache.org/jira/browse/SPARK-8617
Addresses .inprogress files but not event log files that are still running.
Identify a mechanism to set a "max file" size so that the file is rolled over when it reaches this size?
Attachments
Issue Links
- links to
1.
|
Roll over event log files |
|
Resolved | Jungtaek Lim |
2.
|
Enable cleanup old event log files |
|
Closed | Unassigned |
3.
|
Compact old event log files and clean up |
|
Resolved | Jungtaek Lim |
4.
|
Apply compaction of event log to SQL events |
|
Resolved | Jungtaek Lim |
5.
|
Integrate event log compactor into Spark History Server |
|
Resolved | Jungtaek Lim |
6.
|
Different behavior between rolling and non-rolling event log |
|
Resolved | Adam Binford |
7.
|
event log directory(spark-history) filled by large .inprogress files for spark streaming applications |
|
Closed | Unassigned |