Uploaded image for project: 'Apache Hudi'
  1. Apache Hudi
  2. HUDI-5989

Fix the date conversion issue when lazy fetching partition path & file slice for HoodieFileIndex is used

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 0.13.1
    • None

    Description

      For partition filter like date column, spark internally will convert it to day numbers after timestamp 0, but hudi lazy fetching doesn't respect this behavior, this pr tries to fix it.

       

      INFO DataSourceStrategy: Pruning directories with: isnotnull(country#80), isnotnull(par_date#81),(country#80 = ID),(par_date#81=19415)

...

INFO AbstractTableFileSystemView: Building file system view for partition (country=ID/par_date=19415)

       

      This will fix the partition pruning bug introduced in: https://github.com/apache/hudi/pull/6680

       

      Attachments

        Issue Links

          links to

          Web Link GitHub Pull Request #8298

          Activity

            People

              voonhous voon
              voonhous voon
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: