Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-17200

Enable MM2 to replicate topics ending in "internal" suffix

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Open
    • Minor
    • Resolution: Unresolved
    • None
    • None
    • mirrormaker

    Description

      In the current Mirror Maker 2 implementation, topics ending in ".internal" or "-internal" cannot be replicated as they are considered connect/mm2 internal topics. 

      In some cases, users have business topics ending in ".internal" or "-internal" that are excluded from the replication for the same reason. This is because of two things:

      (1) The ReplicationPolicy interface excludes all topics ending in ".internal" or "-internal" from the replication, as they are considered internal connect topics:

      /** Internal topics are never replicated. */
      default boolean isInternalTopic(String topic) {
          boolean isKafkaInternalTopic = topic.startsWith("__") || topic.startsWith(".");
          boolean isDefaultConnectTopic =  topic.endsWith("-internal") ||  topic.endsWith(".internal");
          return isMM2InternalTopic(topic) || isKafkaInternalTopic || isDefaultConnectTopic;
      } 

      (2) The DefaultTopicFilter has the following default exclude regular expression:

      ".*[\\-\\.]internal, .*\\.replica, __.*" 

      The goal of this ticket is to enable the replication of such business topics, while making sure that kafka internal topics are not replicated.

      Attachments

        Issue Links

          Activity

            People

              pmarton Patrik Márton
              pmarton Patrik Márton
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated: