Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-16859

History Server storage information is missing

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Not A Problem
    • 1.6.2, 2.0.0
    • None
    • Spark Core

    Description

      It looks like job history storage tab in history server is broken for completed jobs since 1.6.2.
      More specifically it's broken since SPARK-13845.

      I've fixed for my installation by effectively reverting the above patch (see).

      IMHO, the most straightforward fix would be to implement SparkListenerBlockUpdated serialization to JSON in JsonProtocol making sure it works from ReplayListenerBus.

      The downside will be that it will still work incorrectly with pre patch job histories. But then, it doesn't work since 1.6.2 anyhow.

      PS: I'd really love to have this fixed eventually. But I'm pretty new to Apache Spark and missing hands on Scala experience. So I'd prefer that it be fixed by someone experienced with roadmap vision. If nobody volunteers I'll try to patch myself.

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              Hauer Andrei Ivanov
              Votes:
              4 Vote for this issue
              Watchers:
              16 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: