Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-17019

Expose off-heap memory usage in various places

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Minor
    • Resolution: Fixed
    • 2.0.0
    • 2.2.0
    • Spark Core
    • None

    Description

      With SPARK-13992, Spark supports persisting data into off-heap memory, but the usage of off-heap is not exposed currently, it is not so convenient for user to monitor and profile, so here propose to expose off-heap memory as well as on-heap memory usage in various places:

      1. Spark UI's executor page will display both on-heap and off-heap memory usage.
      2. REST request returns both on-heap and off-heap memory.
      3. Also these two memory usage can be obtained programmatically from SparkListener.

      Attachments

        Activity

          People

            jerryshao Saisai Shao
            jerryshao Saisai Shao
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: