Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-6018

NoSuchMethodError in Spark app is swallowed by YARN AM

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Minor
    • Resolution: Fixed
    • 1.2.0
    • 1.2.2, 1.3.0
    • YARN

    Description

      I discovered this bug while testing 1.3 RC with old 1.2 Spark job that I had. Due to changes in DF and SchemaRDD, my app failed with java.lang.NoSuchMethodError. However, AM was marked as succeeded, and the error was silently swallowed.

      The problem is that pattern matching in Spark AM fails to catch NoSuchMethodError-

      15/02/25 20:13:27 INFO cluster.YarnClusterScheduler: YarnClusterScheduler.postStartHook done
Exception in thread "Driver" scala.MatchError: java.lang.NoSuchMethodError: org.apache.spark.sql.hive.HiveContext.table(Ljava/lang/String;)Lorg/apache/spark/sql/SchemaRDD; (of class java.lang.NoSuchMethodError)
	at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:485)

      Attachments

        Issue Links

          is duplicated by

          Bug - A problem which impairs or prevents the functions of the product. SPARK-6449 Driver OOM results in reported application result SUCCESS

          • Major - Major loss of function.
          • Resolved
          is related to

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-6058 Log the error for the EXIT_EXCEPTION_USER_CLASS exit code

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved
          links to

          Pull request #4773 [Github] Pull Request #4773 (piaozhexiu)

          Activity

            People

              cheolsoo Cheolsoo Park
              cheolsoo Cheolsoo Park
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: