Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-26139 Support passing shuffle metrics to exchange operator
  3. SPARK-26140

Enable custom shuffle metrics implementation in shuffle reader

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 2.4.0
    • 3.0.0
    • Spark Core
    • None

    Description

      The first step to pull the creation of TempShuffleReadMetrics out of shuffle layer, so it can be driven by an external caller. Then we can in SQL execution pass in a special metrics reporter that allows updating ShuffleExchangeExec's metrics.

      Attachments

        Activity

          People

            rxin Reynold Xin
            rxin Reynold Xin
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: