Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-21914

Running examples as tests in SQL builtin function documentation

    XMLWordPrintableJSON

Details

    • Test
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 2.3.0
    • 3.0.0
    • SQL
    • None

    Description

      It looks we have added many examples in ExpressionDescription for builtin functions.
      Actually, if I have seen correctly, we have fixed many examples so far in some minor PRs and sometimes require to add the examples as tests sql and golden files.

      As we have formatted examples in ExpressionDescription.examples - https://github.com/apache/spark/blob/ba327ee54c32b11107793604895bd38559804858/sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/ExpressionDescription.java#L44-L50, and we have `SQLQueryTestSuite`, I think we could run the examples as tests like Python's doctests.

      Rough way I am thinking:

      1. Loads the example in ExpressionDescription.
      2. identify queries by >.
      3. identify the rest of them as the results.
      4. run the examples by reusing SQLQueryTestSuite if possible.
      5. compare the output by reusing SQLQueryTestSuite if possible.

      Advantages of doing this I could think for now:

      • Reduce the number of PRs to fix the examples
      • De-duplicate the test cases that should be added into sql and golden files.
      • Correct documentation with correct examples.
      • Reduce reviewing costs for documentation fix PRs.

      Attachments

        Issue Links

          Activity

            People

              maxgekk Max Gekk
              gurwls223 Hyukjin Kwon
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: