Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-5751

MR app master fails to start in some cases if mapreduce.job.classloader is true

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 2.2.0
    • 2.4.0
    • None
    • None
    • Reviewed

    Description

      If mapreduce.job.classloader is set to true, and the MR client includes a jetty jar in its libjars or job jar, the MR app master fails to start. A typical stack trace we get is as follows:

      java.lang.ClassCastException: org.mortbay.jetty.webapp.WebInfConfiguration cannot be cast to org.mortbay.jetty.webapp.Configuration
	at org.mortbay.jetty.webapp.WebAppContext.loadConfigurations(WebAppContext.java:890)
	at org.mortbay.jetty.webapp.WebAppContext.doStart(WebAppContext.java:462)
	at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
	at org.mortbay.jetty.handler.HandlerCollection.doStart(HandlerCollection.java:152)
	at org.mortbay.jetty.handler.ContextHandlerCollection.doStart(ContextHandlerCollection.java:156)
	at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
	at org.mortbay.jetty.handler.HandlerWrapper.doStart(HandlerWrapper.java:130)
	at org.mortbay.jetty.Server.doStart(Server.java:224)
	at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
	at org.apache.hadoop.http.HttpServer.start(HttpServer.java:676)
	at org.apache.hadoop.yarn.webapp.WebApps$Builder.start(WebApps.java:208)
	at org.apache.hadoop.mapreduce.v2.app.client.MRClientService.start(MRClientService.java:151)
	at org.apache.hadoop.yarn.service.CompositeService.start(CompositeService.java:68)
	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.start(MRAppMaster.java:1040)
	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$1.run(MRAppMaster.java:1307)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:415)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1478)
	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1303)
	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1259)

      This happens because as part of the MR app master start the jetty classes are loaded normally through the app classloader, but WebAppContext tries to load the specific Configuration class via the thread context classloader (which had been set to the user job classloader).

      Attachments

        1. mapreduce-5751.patch
          2 kB
          Sangjin Lee
        2. mapreduce-5751.patch
          1 kB
          Sangjin Lee

        Issue Links

          relates to

          Bug - A problem which impairs or prevents the functions of the product. MAPREDUCE-5957 AM throws ClassNotFoundException with job classloader enabled if custom output format/committer is used

          • Major - Major loss of function.
          • Closed

          Activity

            People

              sjlee0 Sangjin Lee
              sjlee0 Sangjin Lee
              Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: