Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-38075

Hive script transform with order by and limit will return fake rows

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.1.2, 3.2.0, 3.1.3, 3.2.1, 3.3.0
    • 3.1.3, 3.3.0, 3.2.2
    • SQL

    Description

      For example:

      create or replace temp view t as
      select * from values
      (1),
      (2),
      (3)
      as t(a);
      
      select transform(a)
      USING 'cat' AS (a int)
      FROM t order by a limit 10;
      

      This returns:

      NULL
      NULL
      NULL
      1
      2
      3
      

      Without order by and limit, the query returns:

      1
      2
      3
      

      Spark script transform does not have this issue. That is, if spark.sql.catalogImplementation=in-memory, Spark does not return fake rows.

      Attachments

        Activity

          People

            bersprockets Bruce Robbins
            bersprockets Bruce Robbins
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: