Uploaded image for project: 'Oozie'
  1. Oozie
  2. OOZIE-3492

[spark-action] Missing HADOOP_CONF_DIR property

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • trunk
    • 5.2.0
    • action
    • None

    Description

      Spark action fails with the following error:

      Failing Oozie Launcher, When running with master 'yarn-cluster' either HADOOP_CONF_DIR or YARN_CONF_DIR must be set in the environment.
      org.apache.spark.SparkException: When running with master 'yarn-cluster' either HADOOP_CONF_DIR or YARN_CONF_DIR must be set in the environment.
              at org.apache.spark.deploy.SparkSubmitArguments.error(SparkSubmitArguments.scala:657)
              at org.apache.spark.deploy.SparkSubmitArguments.validateSubmitArguments(SparkSubmitArguments.scala:290)
              at org.apache.spark.deploy.SparkSubmitArguments.validateArguments(SparkSubmitArguments.scala:251)
              at org.apache.spark.deploy.SparkSubmitArguments.<init>(SparkSubmitArguments.scala:120)
              at org.apache.spark.deploy.SparkSubmit$$anon$2$$anon$1.<init>(SparkSubmit.scala:926)
              at org.apache.spark.deploy.SparkSubmit$$anon$2.parseArguments(SparkSubmit.scala:926)
              at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:81)
              at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:939)
              at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:948)
              at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
              at org.apache.oozie.action.hadoop.SparkMain.runSpark(SparkMain.java:186)
              at org.apache.oozie.action.hadoop.SparkMain.run(SparkMain.java:93)
              at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:104)
              at org.apache.oozie.action.hadoop.SparkMain.main(SparkMain.java:60)
              at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
              at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
              at java.lang.reflect.Method.invoke(Method.java:498)
              at org.apache.oozie.action.hadoop.LauncherAM.runActionMain(LauncherAM.java:410)
              at org.apache.oozie.action.hadoop.LauncherAM.access$300(LauncherAM.java:55)
              at org.apache.oozie.action.hadoop.LauncherAM$2.run(LauncherAM.java:223)
              at java.security.AccessController.doPrivileged(Native Method)
              at javax.security.auth.Subject.doAs(Subject.java:422)
              at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
              at org.apache.oozie.action.hadoop.LauncherAM.run(LauncherAM.java:217)
              at org.apache.oozie.action.hadoop.LauncherAM$1.run(LauncherAM.java:153)
              at java.security.AccessController.doPrivileged(Native Method)
              at javax.security.auth.Subject.doAs(Subject.java:422)
              at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
              at org.apache.oozie.action.hadoop.LauncherAM.main(LauncherAM.java:141)

      if the followings are true:

      • yarn.nodemanager.env-whitelist does not contain HADOOP_CONF_DIR
      • Hadoop version >= 3.1 (or YARN-7677 is backported)

      Attachments

        1. OOZIE-3492-001.patch
          2 kB
          Andras Salamon

        Issue Links

          Activity

            People

              asalamon74 Andras Salamon
              asalamon74 Andras Salamon
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: