Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
None
Description
MembershipStore and MountTableStore are failing to initialize, logging the following errors on the Router logs:
2022-05-23 16:43:01,156 ERROR org.apache.hadoop.hdfs.server.federation.router.RouterHeartbeatService: Cannot get version for class org.apache.hadoop.hdfs.server.federation.store.MembershipStore org.apache.hadoop.hdfs.server.federation.store.StateStoreUnavailableException: Cached State Store not initialized, MembershipState records not valid at org.apache.hadoop.hdfs.server.federation.store.CachedRecordStore.checkCacheAvailable(CachedRecordStore.java:106) at org.apache.hadoop.hdfs.server.federation.store.CachedRecordStore.getCachedRecords(CachedRecordStore.java:227) at org.apache.hadoop.hdfs.server.federation.router.RouterHeartbeatService.getStateStoreVersion(RouterHeartbeatService.java:131) at org.apache.hadoop.hdfs.server.federation.router.RouterHeartbeatService.updateStateStore(RouterHeartbeatService.java:92) at org.apache.hadoop.hdfs.server.federation.router.RouterHeartbeatService.periodicInvoke(RouterHeartbeatService.java:159) at org.apache.hadoop.hdfs.server.federation.router.PeriodicService$1.run(PeriodicService.java:178) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)
After investigating, we noticed that ZKDelegationTokenSecretManager normally initializes properties for ZooKeeper clients to connect using SASL/Kerberos. If ZKDelegationTokenSecretManager is replaced with a new SecretManager, the SASL properties don't get configured and any StateStores that connect to ZooKeeper fail with the above error.
A potential way to fix this is by setting the JaasConfiguration (currently done in ZKDelegationTokenSecretManager) as part of the StateStoreZooKeeperImpl initialization method.
Attachments
Issue Links
- links to
