Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-31943

SPARK-31500 introduces breaking changes in 2.4.6

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Won't Fix
    • 2.4.6
    • None
    • SQL
    • Patch, Important

    Description

      [31500|https://github.com/apache/spark/pull/28351/files] introduced unimplemented fields of the `Collect` class that cause client extensions of that class to fail unless the spark version is pinned to `2.4.5`. Since this was a minor version bump, seems like this wasn't desired.

      I believe we should be able to at least put default values:

      `convertToBufferElement(value: Any): Any = InternalRow.copyValue(value)`

      `bufferElementType: DataType = child.dataType`

      and restore the `override def eval` in `Collect`, to support compatibility with 2.4.5 while allowing implementers in that file to override them to fix the bug. Since the abstract `Collect` is currently not designed to fix the bug (just provide tools that can be implemented to fix it), this change wouldn't undermine the bug fix, just add backwards compatibility to 2.4.5.

      Attachments

        Activity

          People

            Unassigned Unassigned
            jefferson.v Jefferson V
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: