Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-26797

Start using the new logical types API of Parquet 1.11.0 instead of the deprecated one

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: In Progress
    • Major
    • Resolution: Unresolved
    • 3.0.0
    • None
    • SQL
    • None

    Description

      The 1.11.0 release of parquet-mr will deprecate its logical type API in favour of a newly introduced one. The new API also introduces new subtypes for different timestamp semantics, support for which should be added to Spark in order to read those types correctly.

      At this point only a release candidate of parquet-mr 1.11.0 is available, but that already allows implementing and reviewing this change.

      Attachments

        Issue Links

          is blocked by

          Sub-task - The sub-task of the issue SPARK-26346 Upgrade parquet to 1.11.1

          • Major - Major loss of function.
          • Resolved

          Task - A task that needs to be done. PARQUET-1434 Release parquet-mr 1.11.0

          • Major - Major loss of function.
          • Resolved
          is related to

          Task - A task that needs to be done. SPARK-30905 Execute the TIMESTAMP roadmap

          • Major - Major loss of function.
          • Open
          links to

          Pull request #31685 [Github] Pull Request #31685 (h-vetinari)

          Pull request #31685 [Github] Pull Request #31685 (h-vetinari)

          Web Link GitHub Pull Request #23721

          Activity

            People

              Unassigned Unassigned
              zi Zoltan Ivanfi
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated: