Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-39786

Fix performance gap and inconsistent behavior between using Scala 2.12 and Scala 2.13

    XMLWordPrintableJSON

Details

    • Umbrella
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 3.4.0
    • None
    • Spark Core, SQL
    • None

    Description

      After SPARK-39576, we can run benchmark GitHub Actions to to generate Scala 2.13 benchmark results, so that we can compare with the result of Scala 2.12 one by one to find the performance gap and fix them.

      But if the performance gap is bug of Scala 2.13, we should report it to the Scala community and try to find a workaround way in Spark. If the workaround way not exists due we need to support two Scala versions now, we can left TODO and fix them after Spark only supports Scala 2.13.

      On the other hand, by comparing the UTs or run Scala 2.13 compiled Spark in the production environment, we may find some behavioral differences, they can also be classified as subTasks of the current umbrella.

      Attachments

        Activity

          People

            Unassigned Unassigned
            LuciferYang Yang Jie
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated: