Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.1.0
    • 3.4.0
    • SQL
    • None

    Description

      Spark SQL cannot supports a SQL with nested aggregate as below:

       

      select sum(unique1) FILTER (WHERE
       unique1 IN (SELECT unique1 FROM onek where unique1 < 100)) FROM tenk1;

       

      And Spark will throw exception as follows:

       

      org.apache.spark.sql.AnalysisException
      IN/EXISTS predicate sub-queries can only be used in Filter/Join and a few commands: Aggregate [sum(cast(unique1#x as bigint)) AS sum(unique1)#xL]
      : +- Project [unique1#x]
      : +- Filter (unique1#x < 100)
      : +- SubqueryAlias `onek`
      : +- RelationV2[unique1#x, unique2#x, two#x, four#x, ten#x, twenty#x, hundred#x, thousand#x, twothousand#x, fivethous#x, tenthous#x, odd#x, even#x, stringu1#x, stringu2#x, string4#x] csv file:/home/xitong/code/gengjiaan/spark/sql/core/target/scala-2.12/test-classes/test-data/postgresql/onek.data
      +- SubqueryAlias `tenk1`
       +- RelationV2[unique1#x, unique2#x, two#x, four#x, ten#x, twenty#x, hundred#x, thousand#x, twothousand#x, fivethous#x, tenthous#x, odd#x, even#x, stringu1#x, stringu2#x, string4#x] csv file:/home/xitong/code/gengjiaan/spark/sql/core/target/scala-2.12/test-classes/test-data/postgresql/tenk.data

       

      But PostgreSQL supports this syntax.

      select sum(unique1) FILTER (WHERE
       unique1 IN (SELECT unique1 FROM onek where unique1 < 100)) FROM tenk1;
       sum 
      ------
       4950
      (1 row)

      Attachments

        Activity

          gurwls223 Hyukjin Kwon added a comment - - edited

          beliefer, check you check the comments of its parent JIRA? Should better check other DBMSes too.

          gurwls223 Hyukjin Kwon added a comment - - edited beliefer , check you check the comments of its parent JIRA? Should better check other DBMSes too.
          apachespark Apache Spark added a comment -

          User 'tanelk' has created a pull request for this issue:
          https://github.com/apache/spark/pull/34402

          apachespark Apache Spark added a comment - User 'tanelk' has created a pull request for this issue: https://github.com/apache/spark/pull/34402
          cloud_fan Wenchen Fan added a comment -

          Issue resolved by pull request 34402
          https://github.com/apache/spark/pull/34402

          cloud_fan Wenchen Fan added a comment - Issue resolved by pull request 34402 https://github.com/apache/spark/pull/34402

          People

            tanelk Tanel Kiis
            beliefer Jiaan Geng
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: