Details
Major Description
In a typical router-based federation cluster, hdfs-site.xml is supposed to be:
<property>
<name>dfs.nameservices</name>
<value>ns1,ns2,ns-fed</value>
</property>
<property>
<name>dfs.ha.namenodes.ns-fed</name>
<value>r1,r2</value>
</property>
<property>
<name>dfs.namenode.rpc-address.ns1</name>
<value>host1:8020</value>
</property>
<property>
<name>dfs.namenode.rpc-address.ns2</name>
<value>host2:8020</value>
</property>
<property>
<name>dfs.namenode.rpc-address.ns-fed.r1</name>
<value>host1:8888</value>
</property>
<property>
<name>dfs.namenode.rpc-address.ns-fed.r2</name>
<value>host2:8888</value>
</property>
dfs.ha.namenodes.ns-fed here is used for client to access the Router. However with this configuration on server node, Router fails to start with error:
org.apache.hadoop.HadoopIllegalArgumentException: Configuration has multiple addresses that match local node's address. Please configure the system with dfs.nameservice.id and dfs.ha.namenode.id
at org.apache.hadoop.hdfs.DFSUtil.getSuffixIDs(DFSUtil.java:1198)
at org.apache.hadoop.hdfs.DFSUtil.getNameServiceId(DFSUtil.java:1131)
at org.apache.hadoop.hdfs.DFSUtil.getNamenodeNameServiceId(DFSUtil.java:1086)
at org.apache.hadoop.hdfs.server.federation.router.Router.createLocalNamenodeHearbeatService(Router.java:466)
at org.apache.hadoop.hdfs.server.federation.router.Router.createNamenodeHearbeatServices(Router.java:423)
at org.apache.hadoop.hdfs.server.federation.router.Router.serviceInit(Router.java:199)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
at org.apache.hadoop.hdfs.server.federation.router.DFSRouter.main(DFSRouter.java:69)
2018-03-01 18:05:56,208 ERROR org.apache.hadoop.hdfs.server.federation.router.DFSRouter: Failed to start router
Then the router tries to find the local namenode, multiple properties: dfs.namenode.rpc-address.ns1, dfs.namenode.rpc-address.ns-fed.r1 match the local address.