Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-8870

NullPointerException when glob doesn't return files

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Cannot Reproduce
    • 0.20.205.0, 1.0.3
    • 2.2.0
    • fs, fs/s3
    • None

    Description

      Reading

      s3n://bucket/{a/,b/,c/}

      if one of the globs matches nothing, I get:

      Exception in thread "main" java.lang.NullPointerException
	at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:992)
	at org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:177)
	at org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:208)
	at spark.HadoopRDD.<init>(HadoopRDD.scala:51)
	at spark.SparkContext.hadoopFile(SparkContext.scala:186)
	at spark.SparkContext.textFile(SparkContext.scala:155)
	at com.celtra.analyzer.LogAnalyzer.analyzeSufficientS3Logs(LogAnalyzer.scala:52)
	at com.celtra.analyzer.App$.main(App.scala:164)
	at com.celtra.analyzer.App.main(App.scala)

      I'm not sure whether this is specific to S3 or all filesystems.

      This was occuring in 0.20.205 and I confirmed it's still present in 1.0.3.

      Attachments

        Issue Links

          relates to

          Sub-task - The sub-task of the issue HADOOP-15748 S3 listing inconsistency can raise NPE in globber

          • Major - Major loss of function.
          • Resolved

          Activity

            People

              Unassigned Unassigned
              jakajancar Jaka Jancar
              Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: