Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Not A Problem
-
1.6.3
-
None
Description
Added following to core-site.xml in order to make use of ViewFS in a NameNode federated cluster.
<property> <name>fs.defaultFS</name> <value>viewfs:///</value> </property> <property> <name>fs.viewfs.mounttable.default.link./apps</name> <value>hdfs://nameservice1/apps</value> </property> <property> <name>fs.viewfs.mounttable.default.link./app-logs</name> <value>hdfs://nameservice2/app-logs</value> </property> <property> <name>fs.viewfs.mounttable.default.link./tmp</name> <value>hdfs://nameservice2/tmp</value> </property> <property> <name>fs.viewfs.mounttable.default.link./user</name> <value>hdfs://nameservice2/user</value> </property> <property> <name>fs.viewfs.mounttable.default.link./ns1/user</name> <value>hdfs://nameservice1/user</value> </property> <property> <name>fs.viewfs.mounttable.default.link./ns2/user</name> <value>hdfs://nameservice2/user</value> </property>
Got the following error .
spark-submit --class org.apache.spark.examples.SparkPi --master yarn-client --num-executors 3 --driver-memory 512m --executor-memory 512m --executor-cores 1 ${SPARK_HOME}/lib/spark-examples*.jar 10 18/01/17 02:14:45 INFO spark.SparkContext: Added JAR file:/home/nayakxxxx/hdp26_c4000_stg/spark2/lib/spark-examples_2.11-2.1.1.2.6.2.0-205.jar at spark://xxxxx:35633/jars/spark-examples_2.11-2.1.1.2.6.2.0-205.jar with timestamp 1516155285534 18/01/17 02:14:46 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm2 18/01/17 02:14:46 INFO yarn.Client: Requesting a new application from cluster with 26 NodeManagers 18/01/17 02:14:46 INFO yarn.Client: Verifying our application has not requested more than the maximum memory capability of the cluster (13800 MB per container) 18/01/17 02:14:46 INFO yarn.Client: Will allocate AM container, with 896 MB memory including 384 MB overhead 18/01/17 02:14:46 INFO yarn.Client: Setting up container launch context for our AM 18/01/17 02:14:46 INFO yarn.Client: Setting up the launch environment for our AM container 18/01/17 02:14:46 INFO yarn.Client: Preparing resources for our AM container 18/01/17 02:14:46 INFO security.HDFSCredentialProvider: getting token for namenode: viewfs:/user/nayakxxxx 18/01/17 02:14:46 INFO hdfs.DFSClient: Created HDFS_DELEGATION_TOKEN token 22488202 for nayakxxxx on ha-hdfs:nameservice1 18/01/17 02:14:46 INFO hdfs.DFSClient: Created HDFS_DELEGATION_TOKEN token 50 for nayakxxxx on ha-hdfs:nameservice2 18/01/17 02:14:47 INFO hive.metastore: Trying to connect to metastore with URI thrift://XXXX:9083 18/01/17 02:14:47 INFO hive.metastore: Connected to metastore. 18/01/17 02:14:49 INFO security.HiveCredentialProvider: Get Token from hive metastore: Kind: HIVE_DELEGATION_TOKEN, Service: , Ident: 00 29 6e 61 79 61 6b 6e 69 68 61 72 72 61 30 31 40 53 54 47 32 30 30 30 2e 48 41 44 4f 4f 50 2e 52 41 4b 55 54 45 4e 2e 43 4f 4d 04 68 69 76 65 00 8a 01 61 01 e5 be 03 8a 01 61 25 f2 42 03 8d 02 21 bb 8e 02 b7 18/01/17 02:14:49 WARN yarn.Client: Neither spark.yarn.jars nor spark.yarn.archive is set, falling back to uploading libraries under SPARK_HOME. 18/01/17 02:14:50 INFO yarn.Client: Uploading resource file:/tmp/spark-7498ee81-d22b-426e-9466-3a08f7c827b1/__spark_libs__6643608006679813597.zip -> viewfs:/user/nayakxxxx/.sparkStaging/application_1515035441414_275503/__spark_libs__6643608006679813597.zip 18/01/17 02:14:55 INFO yarn.Client: Uploading resource file:/tmp/spark-7498ee81-d22b-426e-9466-3a08f7c827b1/__spark_conf__405432153902988742.zip -> viewfs:/user/nayakxxxx/.sparkStaging/application_1515035441414_275503/__spark_conf__.zip 18/01/17 02:14:55 INFO spark.SecurityManager: Changing view acls to: nayakxxxx 18/01/17 02:14:55 INFO spark.SecurityManager: Changing modify acls to: nayakxxxx 18/01/17 02:14:55 INFO spark.SecurityManager: Changing view acls groups to: 18/01/17 02:14:55 INFO spark.SecurityManager: Changing modify acls groups to: 18/01/17 02:14:55 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(nayakxxxx); groups with view permissions: Set(); users with modify permissions: Set(nayakxxxx); groups with modify permissions: Set() 18/01/17 02:14:55 INFO yarn.Client: Submitting application application_1515035441414_275503 to ResourceManager 18/01/17 02:14:55 INFO impl.YarnClientImpl: Submitted application application_1515035441414_275503 18/01/17 02:14:55 INFO cluster.SchedulerExtensionServices: Starting Yarn extension services with app application_1515035441414_275503 and attemptId None 18/01/17 02:14:56 INFO yarn.Client: Application report for application_1515035441414_275503 (state: ACCEPTED) 18/01/17 02:14:56 INFO yarn.Client: client token: Token { kind: YARN_CLIENT_TOKEN, service: } diagnostics: AM container is launched, waiting for AM container to Register with RM ApplicationMaster host: N/A ApplicationMaster RPC port: -1 queue: common start time: 1516155295100 final status: UNDEFINED tracking URL: http://XXXX:8088/proxy/application_1515035441414_275503/ user: nayakxxxx 18/01/17 02:14:57 INFO yarn.Client: Application report for application_1515035441414_275503 (state: ACCEPTED) 18/01/17 02:14:58 INFO yarn.Client: Application report for application_1515035441414_275503 (state: ACCEPTED) 18/01/17 02:14:59 INFO yarn.Client: Application report for application_1515035441414_275503 (state: ACCEPTED) 18/01/17 02:15:00 INFO yarn.Client: Application report for application_1515035441414_275503 (state: ACCEPTED) 18/01/17 02:15:01 INFO yarn.Client: Application report for application_1515035441414_275503 (state: ACCEPTED) 18/01/17 02:15:02 INFO yarn.Client: Application report for application_1515035441414_275503 (state: ACCEPTED) 18/01/17 02:15:03 INFO yarn.Client: Application report for application_1515035441414_275503 (state: ACCEPTED) 18/01/17 02:15:04 INFO yarn.Client: Application report for application_1515035441414_275503 (state: ACCEPTED) 18/01/17 02:15:05 INFO yarn.Client: Application report for application_1515035441414_275503 (state: ACCEPTED) 18/01/17 02:15:06 INFO yarn.Client: Application report for application_1515035441414_275503 (state: ACCEPTED) 18/01/17 02:15:07 INFO yarn.Client: Application report for application_1515035441414_275503 (state: ACCEPTED) 18/01/17 02:15:08 INFO yarn.Client: Application report for application_1515035441414_275503 (state: ACCEPTED) 18/01/17 02:15:09 INFO yarn.Client: Application report for application_1515035441414_275503 (state: ACCEPTED) 18/01/17 02:15:10 INFO yarn.Client: Application report for application_1515035441414_275503 (state: ACCEPTED) 18/01/17 02:15:11 INFO yarn.Client: Application report for application_1515035441414_275503 (state: ACCEPTED) 18/01/17 02:15:12 INFO yarn.Client: Application report for application_1515035441414_275503 (state: ACCEPTED) 18/01/17 02:15:13 INFO yarn.Client: Application report for application_1515035441414_275503 (state: ACCEPTED) 18/01/17 02:15:14 INFO yarn.Client: Application report for application_1515035441414_275503 (state: ACCEPTED) 18/01/17 02:15:15 INFO yarn.Client: Application report for application_1515035441414_275503 (state: ACCEPTED) 18/01/17 02:15:16 INFO yarn.Client: Application report for application_1515035441414_275503 (state: ACCEPTED) 18/01/17 02:15:17 INFO yarn.Client: Application report for application_1515035441414_275503 (state: ACCEPTED) 18/01/17 02:15:18 INFO yarn.Client: Application report for application_1515035441414_275503 (state: ACCEPTED) 18/01/17 02:15:19 INFO yarn.Client: Application report for application_1515035441414_275503 (state: ACCEPTED) 18/01/17 02:15:20 INFO yarn.Client: Application report for application_1515035441414_275503 (state: ACCEPTED) 18/01/17 02:15:21 INFO yarn.Client: Application report for application_1515035441414_275503 (state: ACCEPTED) 18/01/17 02:15:22 INFO yarn.Client: Application report for application_1515035441414_275503 (state: ACCEPTED) 18/01/17 02:15:23 INFO yarn.Client: Application report for application_1515035441414_275503 (state: ACCEPTED) 18/01/17 02:15:24 INFO yarn.Client: Application report for application_1515035441414_275503 (state: ACCEPTED) 18/01/17 02:15:25 INFO yarn.Client: Application report for application_1515035441414_275503 (state: ACCEPTED) 18/01/17 02:15:26 INFO yarn.Client: Application report for application_1515035441414_275503 (state: ACCEPTED) 18/01/17 02:15:27 INFO yarn.Client: Application report for application_1515035441414_275503 (state: ACCEPTED) 18/01/17 02:15:28 INFO yarn.Client: Application report for application_1515035441414_275503 (state: ACCEPTED) 18/01/17 02:15:29 INFO yarn.Client: Application report for application_1515035441414_275503 (state: ACCEPTED) 18/01/17 02:15:30 INFO yarn.Client: Application report for application_1515035441414_275503 (state: ACCEPTED) 18/01/17 02:15:31 INFO yarn.Client: Application report for application_1515035441414_275503 (state: ACCEPTED) 18/01/17 02:15:32 INFO yarn.Client: Application report for application_1515035441414_275503 (state: ACCEPTED) 18/01/17 02:15:33 INFO yarn.Client: Application report for application_1515035441414_275503 (state: ACCEPTED) 18/01/17 02:15:34 INFO yarn.Client: Application report for application_1515035441414_275503 (state: FAILED) 18/01/17 02:15:34 INFO yarn.Client: client token: N/A diagnostics: Application application_1515035441414_275503 failed 10 times due to AM Container for appattempt_1515035441414_275503_000010 exited with exitCode: -1000 For more detailed output, check the application tracking page: http://xxxxx:8088/cluster/app/application_1515035441414_275503 Then click on links to logs of each attempt. Diagnostics: ViewFs: Cannot initialize: Empty Mount table in config for viewfs://default/ java.io.IOException: ViewFs: Cannot initialize: Empty Mount table in config for viewfs://default/ at org.apache.hadoop.fs.viewfs.InodeTree.<init>(InodeTree.java:337) at org.apache.hadoop.fs.viewfs.ViewFileSystem$1.<init>(ViewFileSystem.java:168) at org.apache.hadoop.fs.viewfs.ViewFileSystem.initialize(ViewFileSystem.java:168) at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2795) at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:99) at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2829) at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2811) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:390) at org.apache.hadoop.fs.Path.getFileSystem(Path.java:295) at org.apache.hadoop.yarn.util.FSDownload.copy(FSDownload.java:251) at org.apache.hadoop.yarn.util.FSDownload.access$000(FSDownload.java:63) at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:361) at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:359) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866) at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:358) at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:62) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Failing this attempt. Failing the application. ApplicationMaster host: N/A ApplicationMaster RPC port: -1 queue: common start time: 1516155295100 final status: FAILED tracking URL: http://xxxxxx:8088/cluster/app/application_1515035441414_275503 user: nayakxxxx 18/01/17 02:15:34 INFO yarn.Client: Deleted staging directory viewfs:/user/nayakxxxx/.sparkStaging/application_1515035441414_275503 18/01/17 02:15:34 ERROR spark.SparkContext: Error initializing SparkContext.