Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Won't Fix
-
2.4.6
-
None
-
Patch, Important
Description
[31500|https://github.com/apache/spark/pull/28351/files] introduced unimplemented fields of the `Collect` class that cause client extensions of that class to fail unless the spark version is pinned to `2.4.5`. Since this was a minor version bump, seems like this wasn't desired.
I believe we should be able to at least put default values:
`convertToBufferElement(value: Any): Any = InternalRow.copyValue(value)`
`bufferElementType: DataType = child.dataType`
and restore the `override def eval` in `Collect`, to support compatibility with 2.4.5 while allowing implementers in that file to override them to fix the bug. Since the abstract `Collect` is currently not designed to fix the bug (just provide tools that can be implemented to fix it), this change wouldn't undermine the bug fix, just add backwards compatibility to 2.4.5.