Uploaded image for project: 'Apache Hudi'
  1. Apache Hudi
  2. HUDI-5411

Make sure Trino does not re-instantiates Hive's InputFormat for every partition during file listing

    XMLWordPrintableJSON

Details

    Description

      To unblock 0.12.2, we've implemented a stop-gap falling back to FileSystemView-based listing (HUDI-5409).

      This is not an appropriate long-term solution though, and we need to make sure we fix it properly by avoiding re-instantiating InputFormats w/in Trino itself (so that we can properly use the FileIndex and MT) 

      Attachments

        Issue Links

          is related to

          Bug - A problem which impairs or prevents the functions of the product. HUDI-5442 Fix HiveHoodieTableFileIndex to use lazy listing

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. HUDI-6734 Add back HUDI-5409 in Hudi 0.12.x branch

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed
          relates to

          Bug - A problem which impairs or prevents the functions of the product. HUDI-5409 Avoid file index and use fs view cache in COW input format

          • Blocker - Blocks development and/or testing work, production could not run
          • Closed
          links to

          Web Link GitHub Pull Request #7527

          Activity

            People

              codope Sagar Sumit
              alexey.kudinkin Alexey Kudinkin
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: