Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-35808

Let ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG be overridable by user in KafkaSourceBuilder

    XMLWordPrintableJSON

Details

    Description

      This issue is a follow-up to this mailing list discussion

      I'd like to propose letting the ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG be overridable by user in KafkaSourceBuilder, as shown in this DRAFT PR:
       
      https://github.com/apache/flink-connector-kafka/pull/108
       
      From the PR description: 

      This allows users to easily implement the claim check large message pattern without bringing any concerns into the Flink codebase otherwise, by specifying a value.deserializer that handles it, but otherwise passes through the bytes.
      Note: overriding value.serializer is already supported on the Producer side: 

       
      Other Reading:
      https://www.kai-waehner.de/blog/2020/08/07/apache-kafka-handling-large-messages-and-files-for-image-video-audio-processing/

      https://www.conduktor.io/kafka/how-to-send-large-messages-in-apache-kafka/#Option-1:-using-an-external-store-(GB-size-messages)-0

       
      What do folks think? If it seems reasonable I can follow the steps in the contribution guide.  

      Attachments

        Issue Links

          Activity

            People

              klam-shop Kevin Lam
              klam-shop Kevin Lam
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: