Details
-
Improvement
-
Status: In Progress
-
Major
-
Resolution: Unresolved
-
None
-
None
Description
Half-float values are not supported in Parquet. Here is a previous issue that talks about that: https://issues.apache.org/jira/browse/PARQUET-1647
So, this will not work:
import pyarrow as pa import pyarrow.parquet as pq import numpy as np arr = pa.array(np.float16([0.1, 2.2, 3])) table = pa.table({'a': arr}) pq.write_table(table, "test_halffloat.parquet")
This is a proposal to store float16 values in Parquet as FixedSizeBinary, and then restore them to float16 when reading them back in.
Attachments
Issue Links
- depends upon
-
PARQUET-758 [Format] HALF precision FLOAT Logical type
- Resolved
- links to