Details
Description
ExportSnapshot failed when exporting to remote secure kerberos cluster
biadmin@hdtest249:/opt/ibm/biginsights/hbase/lib> /opt/ibm/biginsights/hbase/bin/hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -snapshot s1 -copy-to hdfs://bdvm197.svl.ibm.com:9000/user/biadmin/mysnapshots 14/04/01 14:07:20 INFO util.FSVisitor: No logs under directory:hdfs://hdtest249.svl.ibm.com:9000/hbase/.hbase-snapshot/s1/WALs 14/04/01 14:07:22 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library 14/04/01 14:07:22 INFO compress.CodecPool: Got brand-new compressor [.deflate] 14/04/01 14:07:22 INFO hdfs.DFSClient: Created HDFS_DELEGATION_TOKEN token 17 for biadmin on 9.30.195.103:9000 14/04/01 14:07:22 INFO hdfs.DFSClient: Created HDFS_DELEGATION_TOKEN token 14 for biadmin on 9.30.122.206:9000 14/04/01 14:07:24 INFO hdfs.DFSClient: Created HDFS_DELEGATION_TOKEN token 18 for biadmin on 9.30.195.103:9000 14/04/01 14:07:24 INFO security.TokenCache: Got dt for hdfs://hdtest249.svl.ibm.com:9000/user/biadmin/.staging/job_201404010039_0015/libjars/guava-12.0.1.jar;uri=9.30.195.103:9000;t.service=9.30.195.103:9000 14/04/01 14:07:24 INFO input.FileInputFormat: Total input paths to process : 1 14/04/01 14:07:25 INFO mapred.JobClient: Running job: job_201404010039_0015 14/04/01 14:07:26 INFO mapred.JobClient: map 0% reduce 0% 14/04/01 14:07:41 INFO mapred.JobClient: Task Id : attempt_201404010039_0015_m_000000_0, Status : FAILED java.io.IOException: Failed on local exception: java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]; Host Details : local host is: "hdtest250.svl.ibm.com/9.30.195.104"; destination host is: "bdvm197.svl.ibm.com":9000; Caused by: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS] at org.apache.hadoop.security.SaslRpcClient.selectSaslClient(SaslRpcClient.java:173) at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:390) at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:494) at org.apache.hadoop.ipc.Client$Connection.access$1700(Client.java:314) at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:659) at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:655) at java.security.AccessController.doPrivileged(AccessController.java:366) at javax.security.auth.Subject.doAs(Subject.java:572) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1502) at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:654) ... 31 more
With HBASE-10608, we acquired the tokens, but we didn't add them to the MR job.