Details
-
Sub-task
-
Status: Closed
-
Major
-
Resolution: Duplicate
-
2.0.0
-
None
-
None
Description
Currently there are some code duplicates in continuous Parquet writes (as in Structured Streaming) and non-continuous writes; see ParquetFileFormat#prepareWrite() and ParquetFileFormat#ParquetOutputWriterFactory.
This may lead to inconsistent behavior, when we only change one piece of code but not the other.
Attachments
Issue Links
- duplicates
-
SPARK-17924 Consolidate streaming and batch write path
- Resolved
- links to