Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-28594

Allow event logs for running streaming apps to be rolled over

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.0.0
    • 3.0.0
    • Spark Core

    Description

      At all current Spark releases when event logging on spark streaming is enabled the event logs grow massively.  The files continue to grow until the application is stopped or killed.

      The Spark history server then has difficulty processing the files.

      https://issues.apache.org/jira/browse/SPARK-8617

      Addresses .inprogress files but not event log files that are still running.

      Identify a mechanism to set a "max file" size so that the file is rolled over when it reaches this size?

       

       

      Attachments

        Issue Links

          There are no Sub-Tasks for this issue.

          Activity

            People

              kabhwan Jungtaek Lim
              slevett Stephen Levett
              Felix Cheung Felix Cheung
              Votes:
              2 Vote for this issue
              Watchers:
              16 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: