Uploaded image for project: 'Apache Hudi'
  1. Apache Hudi
  2. HUDI-4472

Revisit schema handling in HoodieSparkSqlWriter

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Blocker
    • Resolution: Fixed
    • 0.12.0
    • 0.13.0
    • writer-core
    • None

    Description

      After many features aimed to bring more and more sophisticated support of schema evolution were layered in w/in HoodieSparkSqlWriter, it's currently requiring careful attention to reconcile many flows and make sure that the original invariants still hold.

       

      One example of the issue was discovered while addressing HUDI-4081 (which was duct-typed in #6213 to avoid substantial changes before the release)

      Attachments

        Issue Links

          contains

          Bug - A problem which impairs or prevents the functions of the product. HUDI-864 parquet schema conflict: optional binary <some-field> (UTF8) is not a group

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed
          is a child of

          Improvement - An improvement or enhancement to an existing feature or task. HUDI-4692 Clean up HoodieSparkSqlWriter

          • Critical - Crashes, loss of data, severe memory leak.
          • Open
          is a parent of

          Bug - A problem which impairs or prevents the functions of the product. HUDI-4588 Ingestion failing if source column is dropped

          • Blocker - Blocks development and/or testing work, production could not run
          • Closed
          is related to

          Improvement - An improvement or enhancement to an existing feature or task. HUDI-4872 Support automatic schema evolution for SQL MERGE INTO with UPDATE */ INESRT *

          • Major - Major loss of function.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. HUDI-4276 Reconcile schema - inject null values for missing fields and add new fields

          • Blocker - Blocks development and/or testing work, production could not run
          • Closed
          relates to

          Improvement - An improvement or enhancement to an existing feature or task. HUDI-4886 Detect incompatible schema change during deltastreamer ingestion

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed
          links to

          Web Link GH PR #6358

          Activity

            People

              alexey.kudinkin Alexey Kudinkin
              alexey.kudinkin Alexey Kudinkin
              Ethan Guo (this is the old account; please use "yihua"), Shiyan Xu, sivabalan narayanan
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: