Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-13267

Number of partition is always 1 for Iceberg tables

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Major
    • Resolution: Unresolved
    • None
    • None
    • Frontend
    • ghx-label-5

    Description

      Impala in general sees the Iceberg tables as unpartitioned and let's the partitioning happen within Iceberg. As a result the query profiles for the SCANs can be misleading when the number of partitions show always 1/1.

      We should either fix this or write 'N/A' or such instead of 1/1.

      Fixing might not be that straightforward because when planning the query Iceberg gives us a list of files to read but we don't know how they are aligned in terms of partitions. So we might have to do the files vs partitions matching ourselves.

      Wondering if we can enhance Iceberg ScanMetrics so that it not just hold the number of files, but the number of partitions too and then we can simply use this metric.

      Attachments

        Activity

          People

            Unassigned Unassigned
            gaborkaszab Gabor Kaszab
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated: