Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-17521

bootstrap-controller option buggy behavior

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 3.7.1
    • None
    • admin
    • None

    Description

      Once running kafka admin tools with --bootstrap-controller, I am experiencing weird behavior. Let me show examples.

      [appuser@e4bbc669d343 ~]$ kafka-configs --describe --bootstrap-controller kafka1:9093 --command-config /tmp/kafka-client.properties --entity-type brokers --entity-name 1
Dynamic configs for broker 1 are:

      That's "sort of" fine, but:

      • my set up consists of 3 controller nodes (1,2,3) and 3 broker nodes (4,5,6).
      • entity-type must be "brokers", even though I am connecting to a controller (9093/tcp is a controller listener)
      • node 1 is not a broker, but a controller instead ("for broker 1 are ...")

      When trying to describe config for node 2:

      [appuser@e4bbc669d343 ~]$ kafka-configs --describe --bootstrap-controller kafka1:9093 --command-config /tmp/kafka-client.properties --entity-type brokers --entity-name 2
Dynamic configs for broker 2 are:
Error while executing config command with args '--describe --bootstrap-controller kafka1:9093 --command-config /tmp/kafka-client.properties --entity-type brokers --entity-name 2'
java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.InvalidRequestException: Unexpected broker id, expected 1 or empty string, but received 2
    at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396)
    at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2096)
    at org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:180)
    at kafka.admin.ConfigCommand$.getResourceConfig(ConfigCommand.scala:610)
    at kafka.admin.ConfigCommand$.$anonfun$describeResourceConfig$5(ConfigCommand.scala:568)
    at kafka.admin.ConfigCommand$.$anonfun$describeResourceConfig$5$adapted(ConfigCommand.scala:560)
    at scala.collection.immutable.List.foreach(List.scala:333)
    at kafka.admin.ConfigCommand$.describeResourceConfig(ConfigCommand.scala:560)
    at kafka.admin.ConfigCommand$.describeConfig(ConfigCommand.scala:538)
    at kafka.admin.ConfigCommand$.processCommand(ConfigCommand.scala:343)
    at kafka.admin.ConfigCommand$.main(ConfigCommand.scala:97)
    at kafka.admin.ConfigCommand.main(ConfigCommand.scala)
Caused by: org.apache.kafka.common.errors.InvalidRequestException: Unexpected broker id, expected 1 or empty string, but received 2

      Ehm, what? Expected 1? I need to describe configs for node 2, not 1. The same thing happens, once connecting to node 2 instead of node 1:

      [appuser@e4bbc669d343 ~]$ kafka-configs --describe --bootstrap-controller kafka2:9093 --command-config /tmp/kafka-client.properties --entity-type brokers --entity-name 2
Dynamic configs for broker 2 are:
Error while executing config command with args '--describe --bootstrap-controller kafka2:9093 --command-config /tmp/kafka-client.properties --entity-type brokers --entity-name 2'
java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.InvalidRequestException: Unexpected broker id, expected 1 or empty string, but received 2
    at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396)
    at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2096)
    at org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:180)
    at kafka.admin.ConfigCommand$.getResourceConfig(ConfigCommand.scala:610)
    at kafka.admin.ConfigCommand$.$anonfun$describeResourceConfig$5(ConfigCommand.scala:568)
    at kafka.admin.ConfigCommand$.$anonfun$describeResourceConfig$5$adapted(ConfigCommand.scala:560)
    at scala.collection.immutable.List.foreach(List.scala:333)
    at kafka.admin.ConfigCommand$.describeResourceConfig(ConfigCommand.scala:560)
    at kafka.admin.ConfigCommand$.describeConfig(ConfigCommand.scala:538)
    at kafka.admin.ConfigCommand$.processCommand(ConfigCommand.scala:343)
    at kafka.admin.ConfigCommand$.main(ConfigCommand.scala:97)
    at kafka.admin.ConfigCommand.main(ConfigCommand.scala)
Caused by: org.apache.kafka.common.errors.InvalidRequestException: Unexpected broker id, expected 1 or empty string, but received 2

      If i specify --all instead of entity-name, what I see is:

      [appuser@e4bbc669d343 ~]$ kafka-configs --describe --bootstrap-controller kafka2:9093 --command-config /tmp/kafka-client.properties --entity-type brokers --all
All configs for broker 1 are:
  advertised.listeners=null sensitive=false synonyms={}
<redacted>
  zookeeper.ssl.truststore.type=null sensitive=false synonyms={}
All configs for broker 2 are:
Error while executing config command with args '--describe --bootstrap-controller kafka2:9093 --command-config /tmp/kafka-client.properties --entity-type brokers --all'
java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.InvalidRequestException: Unexpected broker id, expected 1 or empty string, but received 2
    at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396)
    at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2096)
    at org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:180)
    at kafka.admin.ConfigCommand$.getResourceConfig(ConfigCommand.scala:610)
    at kafka.admin.ConfigCommand$.$anonfun$describeResourceConfig$5(ConfigCommand.scala:568)
    at kafka.admin.ConfigCommand$.$anonfun$describeResourceConfig$5$adapted(ConfigCommand.scala:560)
    at scala.collection.immutable.List.foreach(List.scala:333)
    at kafka.admin.ConfigCommand$.describeResourceConfig(ConfigCommand.scala:560)
    at kafka.admin.ConfigCommand$.describeConfig(ConfigCommand.scala:538)
    at kafka.admin.ConfigCommand$.processCommand(ConfigCommand.scala:343)
    at kafka.admin.ConfigCommand$.main(ConfigCommand.scala:97)
    at kafka.admin.ConfigCommand.main(ConfigCommand.scala)
Caused by: org.apache.kafka.common.errors.InvalidRequestException: Unexpected broker id, expected 1 or empty string, but received 2

      Where exactly did I specify "2"?

      If I want to describe configs for node 4 (broker), no matter what node I use as --bootstrap-controller, I get a timeout:

      [appuser@e4bbc669d343 ~]$ kafka-configs --describe --bootstrap-controller kafka3:9093 --command-config /tmp/kafka-client.properties --entity-type brokers --entity-name 4
Dynamic configs for broker 4 are:
Error while executing config command with args '--describe --bootstrap-controller kafka3:9093 --command-config /tmp/kafka-client.properties --entity-type brokers --entity-name 4'
java.util.concurrent.TimeoutException
    at java.base/java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1960)
    at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2095)
    at org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:180)
    at kafka.admin.ConfigCommand$.getResourceConfig(ConfigCommand.scala:610)
    at kafka.admin.ConfigCommand$.$anonfun$describeResourceConfig$5(ConfigCommand.scala:568)
    at kafka.admin.ConfigCommand$.$anonfun$describeResourceConfig$5$adapted(ConfigCommand.scala:560)
    at scala.collection.immutable.List.foreach(List.scala:333)
    at kafka.admin.ConfigCommand$.describeResourceConfig(ConfigCommand.scala:560)
    at kafka.admin.ConfigCommand$.describeConfig(ConfigCommand.scala:538)
    at kafka.admin.ConfigCommand$.processCommand(ConfigCommand.scala:343)
    at kafka.admin.ConfigCommand$.main(ConfigCommand.scala:97)
    at kafka.admin.ConfigCommand.main(ConfigCommand.scala)

      If I specify any of the bootstrap servers, it works fine for brokers:

      [appuser@e4bbc669d343 ~]$ kafka-configs --describe --bootstrap-server kafka4:9092 --entity-type brokers --entity-name 4
Dynamic configs for broker 4 are:
[appuser@e4bbc669d343 ~]$ kafka-configs --describe --bootstrap-server kafka4:9092 --entity-type brokers --entity-name 5
Dynamic configs for broker 5 are:
[appuser@e4bbc669d343 ~]$ kafka-configs --describe --bootstrap-server kafka4:9092 --entity-type brokers --entity-name 6
Dynamic configs for broker 6 are:
[appuser@e4bbc669d343 ~]$

      but describing config for controller node(s) fails:

      [appuser@e4bbc669d343 ~]$ kafka-configs --describe --bootstrap-server kafka4:9092 --entity-type brokers --entity-name 1
Dynamic configs for broker 1 are:
Error while executing config command with args '--describe --bootstrap-server kafka4:9092 --entity-type brokers --entity-name 1'
java.util.concurrent.TimeoutException
    at java.base/java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1960)
    at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2095)
    at org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:180)
    at kafka.admin.ConfigCommand$.getResourceConfig(ConfigCommand.scala:610)
    at kafka.admin.ConfigCommand$.$anonfun$describeResourceConfig$5(ConfigCommand.scala:568)
    at kafka.admin.ConfigCommand$.$anonfun$describeResourceConfig$5$adapted(ConfigCommand.scala:560)
    at scala.collection.immutable.List.foreach(List.scala:333)
    at kafka.admin.ConfigCommand$.describeResourceConfig(ConfigCommand.scala:560)
    at kafka.admin.ConfigCommand$.describeConfig(ConfigCommand.scala:538)
    at kafka.admin.ConfigCommand$.processCommand(ConfigCommand.scala:343)
    at kafka.admin.ConfigCommand$.main(ConfigCommand.scala:97)
    at kafka.admin.ConfigCommand.main(ConfigCommand.scala)
[appuser@e4bbc669d343 ~]$

      And the last stuff to be reported today is a SSL handshake problem for kafka-features:

      [appuser@e4bbc669d343 ~]$ kafka-features --bootstrap-controller kafka1:9093 --command-config /tmp/kafka-client.properties describe
[2024-09-11 10:13:42,466] ERROR [AdminClient clientId=adminclient-1] Connection to node 2 (/127.0.0.1:9093) failed authentication due to: SSL handshake failed (org.apache.kafka.clients.NetworkClient)
[2024-09-11 10:13:42,467] WARN [AdminClient clientId=adminclient-1] Metadata update failed due to authentication error (org.apache.kafka.clients.admin.internals.AdminMetadataManager)
org.apache.kafka.common.errors.SslAuthenticationException: SSL handshake failed
Caused by: javax.net.ssl.SSLHandshakeException: No subject alternative names matching IP address 127.0.0.1 found
    at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:131)
    at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:378)
    at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:321)
    at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:316)
    at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1351)
    at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.onConsumeCertificate(CertificateMessage.java:1226)
    at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.consume(CertificateMessage.java:1169)
    at java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:396)
    at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:480)
    at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1277)
    at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1264)
    at java.base/java.security.AccessController.doPrivileged(AccessController.java:712)
    at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:1209)
    at org.apache.kafka.common.network.SslTransportLayer.runDelegatedTasks(SslTransportLayer.java:443)
    at org.apache.kafka.common.network.SslTransportLayer.handshakeUnwrap(SslTransportLayer.java:532)
    at org.apache.kafka.common.network.SslTransportLayer.doHandshake(SslTransportLayer.java:381)
    at org.apache.kafka.common.network.SslTransportLayer.handshake(SslTransportLayer.java:301)
    at org.apache.kafka.common.network.KafkaChannel.prepare(KafkaChannel.java:178)
    at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:543)
    at org.apache.kafka.common.network.Selector.poll(Selector.java:481)
    at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:585)
    at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.processRequests(KafkaAdminClient.java:1504)
    at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.run(KafkaAdminClient.java:1435)
    at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: java.security.cert.CertificateException: No subject alternative names matching IP address 127.0.0.1 found
    at java.base/sun.security.util.HostnameChecker.matchIP(HostnameChecker.java:165)
    at java.base/sun.security.util.HostnameChecker.match(HostnameChecker.java:101)
    at java.base/sun.security.ssl.X509TrustManagerImpl.checkIdentity(X509TrustManagerImpl.java:458)
    at java.base/sun.security.ssl.X509TrustManagerImpl.checkIdentity(X509TrustManagerImpl.java:432)
    at java.base/sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:292)
    at java.base/sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:144)
    at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1329)
    ... 19 more
org.apache.kafka.common.errors.SslAuthenticationException: SSL handshake failed
[appuser@e4bbc669d343 ~]${code}
The weird thing here would be, why the error is reporting that 127.0.0.1 is missing in AltNames ... while I am using FQDNS:
{code:java}
[appuser@e4bbc669d343 ~]$ ping -c1 kafka1
PING kafka1 (172.30.0.2) 56(84) bytes of data.
64 bytes from e4bbc669d343 (172.30.0.2): icmp_seq=1 ttl=64 time=0.108 ms--- kafka1 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.108/0.108/0.108/0.000 ms

      This being AltNames in the certificate:

                  X509v3 Subject Alternative Name:
                DNS:kafka1.cd4460cf-3d86-4f1b-ad25-a7ec66cecbb8, DNS:kafka1

      None of the names or config refering to localhost, no dns aliases ...

      [appuser@e4bbc669d343 ~]$ cat /etc/hosts
127.0.0.1    localhost
::1    localhost ip6-localhost ip6-loopback
fe00::0    ip6-localnet
ff00::0    ip6-mcastprefix
ff02::1    ip6-allnodes
ff02::2    ip6-allrouters
172.30.0.2    e4bbc669d343

       

      Finally, let me tell you what is my configuration for controllers (1,2,3, the snippet shows config for node 1, other nodes just having respective numbers:

      listener.name.controller.ssl.truststore.type=PEM
listener.name.controller.ssl.keystore.type=PEM
listener.name.controller.ssl.keystore.certificate.chain=<redacted>
transaction.state.log.min.isr=1
process.roles=controller
controller.listener.names=CONTROLLER
group.initial.rebalance.delay.ms=0
controller.quorum.voters=1@kafka1:9093,2@kafka2:9093,3@kafka3:9093
listener.name.controller.ssl.keystore.key=<redacted>
node.id=1
listener.name.controller.ssl.client.auth=required
kraft.mode=true
listener.name.controller.ssl.truststore.certificates=<redacted> listener.security.protocol.map=CONTROLLER:SSL
listener.name.controller.ssl.endpoint.identification.algorithm=https
transaction.state.log.replication.factor=1
listeners=CONTROLLER://0.0.0.0:9093
zookeeper.connect=
log.dirs=/var/lib/kafka/data-1
offsets.topic.replication.factor=1

      ... and brokers:

      listener.name.controller.ssl.truststore.type=PEM
listener.name.controller.ssl.keystore.type=PEM
listener.name.controller.ssl.keystore.certificate.chain=<redacted>
transaction.state.log.min.isr=1
process.roles=broker
controller.listener.names=CONTROLLER
group.initial.rebalance.delay.ms=0
controller.quorum.voters=1@kafka1:9093,2@kafka2:9093,3@kafka3:9093
listener.name.controller.ssl.keystore.key=<redacted>
node.id=4
listener.name.controller.ssl.client.auth=required
advertised.listeners=PLAINTEXT://kafka4:9092
kraft.mode=true
listener.name.controller.ssl.truststore.certificates=<redacted> listener.security.protocol.map=PLAINTEXT:PLAINTEXT,CONTROLLER:SSL
listener.name.controller.ssl.endpoint.identification.algorithm=https
transaction.state.log.replication.factor=1
listeners=PLAINTEXT://0.0.0.0:9092
zookeeper.connect=
log.dirs=/var/lib/kafka/data-4
offsets.topic.replication.factor=1

      ... with respective numbers (5,6) for another two instances.

      Attachments

        Activity

          People

            Unassigned Unassigned
            misko Michal Medvecky
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated: