Description
The option of DataFrameWriterV2 should be passed to storage properties if fallback to v1, to support something such as compressed formats, example:
spark.range(0, 100).writeTo("t1").option("compression", "zstd").using("parquet").create
before
gen: part-00000-644a65ed-0e7a-43d5-8d30-b610a0fb19dc-c000.snappy.parquet
after
gen: part-00000-6eb9d1ae-8fdb-4428-aea3-bd6553954cdd-c000.zstd.parquet ...
Attachments
Issue Links
- links to