Uploaded image for project: 'Apache Hudi'
  1. Apache Hudi
  2. HUDI-6684

Follow up/ fix missing records from bloom filter partition in MDT

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Open
    • Major
    • Resolution: Unresolved
    • None
    • None
    • metadata
    • None

    Description

      As of now, if a bloom filter for a file is missing from bloom filter partition in MDT, we ignore it. 

      HoodieMetadataTableUtil

        // If reading the bloom filter failed then do not add a record for this file
        if (bloomFilterBuffer == null) {
          LOG.error("Failed to read bloom filter from " + addedFilePath);
          return Stream.<HoodieRecord>empty().iterator();
        }
      } 

      we should think about on what scenario, this is possible and how exactly we can handle such situations. 

       

      Attachments

        Activity

          People

            Unassigned Unassigned
            shivnarayan sivabalan narayanan
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated: