Uploaded image for project: 'Ambari'
  1. Ambari
  2. AMBARI-22525

ATS start failed during patch upgrade due to CNF SparkATSPlugin

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Blocker
    • Resolution: Fixed
    • 2.6.1
    • 2.6.1
    • None
    • None

    Description

      STR

      1. Deployed cluster with Ambari 2.5.2.0-298 and HDP version: 2.6.2.0-205
      2. Upgrade Ambari to Target Version: 2.6.1.0-41 | Hash: ee1023b111b2296588d67ecbcd50035f9b525a1c
      3. Perform a full stack express upgrade to HDP-2.6.4.0
      4. Perform a 4th digit patch upgrade to 2.6.4.1-1 with few services (see attached for complete list)
      5. Perform another 4th digit PU to 2.6.4.1-2 with another set of services which now includes Yarn, but not Spark

      Result:
      Post the second PU, ATS fails to start. Logs convey the following:

      2017-11-20 11:36:45,457 FATAL applicationhistoryservice.ApplicationHistoryServer (ApplicationHistoryServer.java:launchAppHistoryServer(177)) - Error starting ApplicationHistoryServer
java.lang.RuntimeException: No class defined for org.apache.spark.deploy.history.yarn.plugin.SparkATSPlugin
        at org.apache.hadoop.yarn.server.timeline.EntityGroupFSTimelineStore.loadPlugIns(EntityGroupFSTimelineStore.java:256)
        at org.apache.hadoop.yarn.server.timeline.EntityGroupFSTimelineStore.serviceInit(EntityGroupFSTimelineStore.java:196)
        at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
        at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107)
        at org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer.serviceInit(ApplicationHistoryServer.java:111)
        at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
        at org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer.launchAppHistoryServer(ApplicationHistoryServer.java:174)
        at org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer.main(ApplicationHistoryServer.java:184)
Caused by: java.lang.ClassNotFoundException: org.apache.spark.deploy.history.yarn.plugin.SparkATSPlugin
        at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        at org.apache.hadoop.util.ApplicationClassLoader.loadClass(ApplicationClassLoader.java:197)
        at org.apache.hadoop.util.ApplicationClassLoader.loadClass(ApplicationClassLoader.java:165)
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:348)
        at org.apache.hadoop.yarn.server.timeline.EntityGroupFSTimelineStore.loadPlugIns(EntityGroupFSTimelineStore.java:243)
        ... 7 more
2017-11-20 11:36:45,460 INFO  util.ExitUtil (ExitUtil.java:terminate(124)) - Exiting with status -1
2017-11-20 11:36:45,465 INFO  applicationhistoryservice.ApplicationHistoryServer (LogAdapter.java:info(45)) - SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down ApplicationHistoryServer at ctr-e134-1499953498516-323313-01-000002.hwx.site/172.27.75.66
************************************************************/

      Looks like the issue is because Spark is not upgraded yet as part of the PU

      Attachments

        1. AMBARI-22525.patch
          10 kB
          Jonathan Hurley

        Issue Links

          links to

          Web Link ReviewBoard

          Activity

            People

              jonathanhurley Jonathan Hurley
              shavi71 Vivek Sharma
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: