Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-33817

Allow ReadDefaultValues = False for non primitive types on Proto3

    XMLWordPrintableJSON

Details

    • Hide
      Protobuf format now supports set 'protobuf.read-default-values' = 'false' for non-primitive types on pb3. Since the default value for 'protobuf.read-default-values' is 'false', this is a breaking change, if you are using pb3, you can set 'protobuf.read-default-values' = 'true' to keep the original behavior.
      Show
      Protobuf format now supports set 'protobuf.read-default-values' = 'false' for non-primitive types on pb3. Since the default value for 'protobuf.read-default-values' is 'false', this is a breaking change, if you are using pb3, you can set 'protobuf.read-default-values' = 'true' to keep the original behavior.

    Description

      Background

       

      The current Protobuf format implementation always sets ReadDefaultValues=False when using Proto3 version. This can cause severe performance degradation for large Protobuf schemas with OneOf fields as the entire generated code needs to be executed during deserialization even when certain fields are not present in the data to be deserialized and all the subsequent nested Fields can be skipped. Proto3 supports hasXXX() methods for checking field presence for non primitive types since Proto version 3.15. In the internal performance benchmarks in our company, we've seen almost 10x difference in performance for one of our real production usecase when allowing to set ReadDefaultValues=False with proto3 version. The exact difference in performance depends on the schema complexity and data payload but we should allow user to set readDefaultValue=False in general.

       

      Solution

       

      Support using ReadDefaultValues=False when using Proto3 version. We need to be careful to check for field presence only on non-primitive types if ReadDefaultValues is false and version used is Proto3

      Attachments

        Issue Links

          Activity

            People

              dsaisharath Sai Sharath Dandi
              dsaisharath Sai Sharath Dandi
              Votes:
              1 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: