Details
-
Improvement
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
4.0.0
Description
From ARROW-10882/https://github.com/apache/arrow/pull/9802
- Backing an InMemoryDataset with a reader is misleading. Let's split that out into a separate class.
- Dataset scanning can then use an I/O thread for the new class. (Note that for Python, we'll need to be careful to release the GIL before any operations so that the I/O thread can acquire the GIL to call into the underlying Python reader/file object.)
- Longer-term, we should interface with Python's async.
Attachments
Issue Links
- relates to
-
ARROW-10882 [Python][Dataset] Writing dataset from python iterator of record batches
- Resolved
- links to