Details
-
Bug
-
Status: Closed
-
Major
-
Resolution: Fixed
-
None
-
None
Description
Custom added Jars in LocalFs throws FNF:
ERROR : Job Submission failed with exception 'java.io.FileNotFoundException(File does not exist: hdfs://localhost:9000/Users/ayushsaxena/code/hive-os/hive/packaging/target/apache-hive-4.0.0-alpha-2-SNAPSHOT-bin/apache-hive-4.0.0-alpha-2-SNAPSHOT-bin/lib/hive-iceberg-handler-4.0.0-alpha-2-SNAPSHOT.jar)' java.io.FileNotFoundException: File does not exist: hdfs://localhost:9000/Users/ayushsaxena/code/hive-os/hive/packaging/target/apache-hive-4.0.0-alpha-2-SNAPSHOT-bin/apache-hive-4.0.0-alpha-2-SNAPSHOT-bin/lib/hive-iceberg-handler-4.0.0-alpha-2-SNAPSHOT.jar at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1756) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1749) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1764) at org.apache.hadoop.fs.FileSystem.resolvePath(FileSystem.java:958) at org.apache.hadoop.mapreduce.v2.util.MRApps.addToClasspathIfNotJar(MRApps.java:342) at org.apache.hadoop.mapreduce.v2.util.MRApps.setClasspath(MRApps.java:275) at org.apache.hadoop.mapred.YARNRunner.setupContainerLaunchContextForAM(YARNRunner.java:525) at org.apache.hadoop.mapred.YARNRunner.createApplicationSubmissionContext(YARNRunner.java:584) at org.apache.hadoop.mapred.YARNRunner.submitJob(YARNRunner.java:326) at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:251) at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1571) at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1568) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1878) at org.apache.hadoop.mapreduce.Job.submit(Job.java:1568) at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:576) at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:571) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1878) at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:571) at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:562) at org.apache.hadoop.hive.ql.exec.mr.ExecDriver.execute(ExecDriver.java:416) at org.apache.hadoop.hive.ql.exec.mr.MapRedTask.execute(MapRedTask.java:158)
Some Applications do consider every Jar Path to be in LocalFileSystem but rest follow the standard Hadoop practice to check the Fs.DefaultFs. We should better make the path qualified
Attachments
Issue Links
- links to