Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-8599 Improve non-deterministic expression handling
  3. SPARK-9083

If order by clause has non-deterministic expressions, we should add a project to materialize results of these expressions

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 1.5.0
    • SQL
    • None

    Description

      When a order by clause has a non-deterministic expression, we actually evaluate it twice, once in the exchange operator when we try to figure out the range partitioner's boundaries and once in the sort operator. We should use a project to materialize the result first.

      Attachments

        Issue Links

          Activity

            People

              cloud_fan Wenchen Fan
              yhuai Yin Huai
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: