Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-6758

Default "" consumer group tracks committed offsets, but is otherwise not a real group

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Duplicate
    • 0.11.0.2
    • None
    • clients
    • None

    Description

      To reproduce:

      • Use the default config for `group.id` of "" (the empty string)
      • Use the default config for `enable.auto.commit` of `true`
      • Use manually assigned partitions with `assign`

      Actual (unexpected) behaviour:

      Consumer offsets are stored for the "" group. Example:

      ~ $ /opt/kafka/kafka_2.11-0.11.0.2/bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group ""
      Note: This will only show information about consumers that use the Java consumer API (non-ZooKeeper-based consumers).

      Consumer group '' has no active members.

      TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID
      my_topic 54 7859593 7865082 5489 - - -
      my_topic 5 14252813 14266419 13606 - - -
      my_topic 39 19099099 19122441 23342 - - -
      my_topic 43 16434573 16449180 14607 - - -.

      ....

       

      However, the "" is not a real group. It doesn't show up with:

      ~ $ /opt/kafka/kafka_2.11-0.11.0.2/bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --list

      You also can't do dynamic partition assignment with it - if you try to `subscribe` when using the default "" group ID, you get:

      AbstractCoordinator: Attempt to join group  failed due to fatal error: The configured groupId is invalid

      Better behaviours:

      (any of these would be preferable, in my opinion)

      • Don't commit offsets with the "" group, and log a warning telling the user that `enable.auto.commit = true` is meaningless in this situation. This is what I would have expected.
      • Don't have a default `group.id`. Some of my reading indicates that the new consumer basically needs a `group.id` to function. If so, force users to choose a group ID so that they're more aware of what will happen.
      • Have a default `group.id` of `default`, and make it a real consumer group. That is, it shows up in lists of groups, it has dynamic partitioning, etc.

      As a user, when I don't set `group.id` I expect that I'm not using consumer groups. This is confirmed to me by listing the consumer groups on the broker and not seeing anything. Therefore, I expect that there will be no offset tracking in Kafka.

      In my specific application, I was wanting `auto.offset.reset` to kick in so that a failed consumer would start at the `latest` offset. However, it started at this unexpectedly stored offset instead.

       

      Attachments

        Issue Links

          Activity

            People

              enether Stanislav Kozlovski
              dwvangeest David van Geest
              Votes:
              1 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: