Details
-
Bug
-
Status: Open
-
Major
-
Resolution: Unresolved
-
2.7.1, 3.3.1
-
None
-
None
Description
When setting Zookeeper to work with SASL plain and Kafka to connect to zookeeper with SASL and Java has FIPS enabled on a FIPS enabled Redhat machine
Kafka cannot connect with zookeeper
Used Apache Kafka 3.3.1 for both zookeeper and client
Operating system Redhat 8
Java 11 installed using yum install java-11-openjdk
set fips mode using command as root
fips-mode-setup --enable
rebooted machine using
systemctl reboot
verified fips mode with
fips-mode-setup --check
set JAVA_HOME to /usr/lib/jvm/jre-11
in config/zookeeper.properties, added properties
admin.enableServer = false
authProvider.1 = org.apache.zookeeper.server.auth.SASLAuthenticationProvider
requireClientAuthScheme = sasl
in config/server.properties, added properties
advertised.listeners = SASL_PLAINTEXT://localhost:9092
listeners = SASL_PLAINTEXT://:9092
security.inter.broker.protocol = SASL_PLAINTEXT
sasl.mechanism.inter.broker.protocol = PLAIN
sasl.enabled.mechanisms = PLAIN
super.users = User:admin
$HOME/zookeeper_jaas.conf value:
Server
$HOME/server_jaas.conf value:
KafkaServer { org.apache.kafka.common.security.plain.PlainLoginModule required username="admin" password="0x572f372b5da34874a3fdf4d8002f" user_admin="0x572f372b5da34874a3fdf4d8002f"; }
;
Client
;
for zookeeper, in kafka directory ran
setenv JAVA_HOME /usr/lib/jvm/jre-11
setenv KAFKA_OPTS -Djava.security.auth.login.config=$HOME/zookeeper_jaas.conf
./bin/zookeeper-server-start.sh ./config/zookeeper.properties
for kafka, in kafka directory ran
setenv JAVA_HOME /usr/lib/jvm/jre-11
setenv KAFKA_OPTS -Djava.security.auth.login.config=$HOME/server_jaas.conf
./bin/kafka-server-start.sh ./config/server.properties
Kafka exits with SASL error
[2022-11-15 13:21:04,344] ERROR SASL authentication with Zookeeper Quorum member failed. (org.apache.zookeeper.ClientCnxn)
javax.security.sasl.SaslException: saslClient failed to initialize properly: it's null.
at org.apache.zookeeper.client.ZooKeeperSaslClient.initialize(ZooKeeperSaslClient.java:399)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1220)
[2022-11-15 13:21:04,348] INFO EventThread shut down for session: 0x1000575ab110000 (org.apache.zookeeper.ClientCnxn)
[2022-11-15 13:21:04,348] INFO [ZooKeeperClient Kafka server] Closing. (kafka.zookeeper.ZooKeeperClient)
[2022-11-15 13:21:04,351] INFO [ZooKeeperClient Kafka server] Closed. (kafka.zookeeper.ZooKeeperClient)
[2022-11-15 13:21:04,355] ERROR Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
kafka.zookeeper.ZooKeeperClientAuthFailedException: Auth failed either before or while waiting for connection
at kafka.zookeeper.ZooKeeperClient.waitUntilConnected(ZooKeeperClient.scala:260)
at kafka.zookeeper.ZooKeeperClient.<init>(ZooKeeperClient.scala:108)
at kafka.zk.KafkaZkClient$.apply(KafkaZkClient.scala:1980)
at kafka.server.KafkaServer.initZkClient(KafkaServer.scala:503)
at kafka.server.KafkaServer.startup(KafkaServer.scala:203)
at kafka.Kafka$.main(Kafka.scala:109)
at kafka.Kafka.main(Kafka.scala)
[2022-11-15 13:21:04,356] INFO shutting down (kafka.server.KafkaServer)
[2022-11-15 13:21:04,363] INFO App info kafka.server for 0 unregistered (org.apache.kafka.common.utils.AppInfoParser)
[2022-11-15 13:21:04,363] INFO shut down completed (kafka.server.KafkaServer)
[2022-11-15 13:21:04,363] ERROR Exiting Kafka due to fatal exception during startup. (kafka.Kafka$)
kafka.zookeeper.ZooKeeperClientAuthFailedException: Auth failed either before or while waiting for connection
at kafka.zookeeper.ZooKeeperClient.waitUntilConnected(ZooKeeperClient.scala:260)
at kafka.zookeeper.ZooKeeperClient.<init>(ZooKeeperClient.scala:108)
at kafka.zk.KafkaZkClient$.apply(KafkaZkClient.scala:1980)
at kafka.server.KafkaServer.initZkClient(KafkaServer.scala:503)
at kafka.server.KafkaServer.startup(KafkaServer.scala:203)
at kafka.Kafka$.main(Kafka.scala:109)
at kafka.Kafka.main(Kafka.scala)
[2022-11-15 13:21:04,368] INFO shutting down (kafka.server.KafkaServer)
When removing FIPS by changing the java conf/security/java.security to have
security.useSystemPropertiesFile=false
The problem does not happen but this property disables FIPS