Details
-
Bug
-
Status: Open
-
Major
-
Resolution: Unresolved
-
None
-
None
-
ghx-label-5
Description
Impala in general sees the Iceberg tables as unpartitioned and let's the partitioning happen within Iceberg. As a result the query profiles for the SCANs can be misleading when the number of partitions show always 1/1.
We should either fix this or write 'N/A' or such instead of 1/1.
Fixing might not be that straightforward because when planning the query Iceberg gives us a list of files to read but we don't know how they are aligned in terms of partitions. So we might have to do the files vs partitions matching ourselves.
Wondering if we can enhance Iceberg ScanMetrics so that it not just hold the number of files, but the number of partitions too and then we can simply use this metric.