Details
-
Bug
-
Status: Resolved
-
Blocker
-
Resolution: Fixed
-
2.4.0
Description
After the Falcon-Atlas hook has been enabled, the following properties are added.
startup.properties
*.application.services=org.apache.falcon.security.AuthenticationInitializationService,\ org.apache.falcon.workflow.WorkflowJobEndNotificationService, \ org.apache.falcon.service.ProcessSubscriberService,\ org.apache.falcon.extensions.ExtensionService,\ org.apache.falcon.service.LifecyclePolicyMap,\ org.apache.falcon.entity.store.ConfigurationStore,\ org.apache.falcon.rerun.service.RetryService,\ org.apache.falcon.rerun.service.LateRunService,\ org.apache.falcon.service.LogCleanupService,\ org.apache.falcon.metadata.MetadataMappingService,\ org.apache.atlas.falcon.service.AtlasService
falcon-env.sh
# Add the Atlas Falcon hook to the Falcon classpath export FALCON_EXTRA_CLASS_PATH=/usr/hdp/current/atlas-client/hook/falcon/*:${FALCON_EXTRA_CLASS_PATH}
Whenever Oozie submits Hive actions, they fail and the application logs show
hive.exec.post.hooks Class not found:org.apache.atlas.hive.hook.HiveHook FAILED: Hive Internal Error: java.lang.ClassNotFoundException(org.apache.atlas.hive.hook.HiveHook) java.lang.ClassNotFoundException: org.apache.atlas.hive.hook.HiveHook at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:348) at org.apache.hadoop.hive.ql.hooks.HookUtils.getHooks(HookUtils.java:60) at org.apache.hadoop.hive.ql.Driver.getHooks(Driver.java:1384) at org.apache.hadoop.hive.ql.Driver.getHooks(Driver.java:1368) at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1595) at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1289) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1156) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1146) at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:216) at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:168) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:379) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:314) at org.apache.hadoop.hive.cli.CliDriver.processReader(CliDriver.java:412) at org.apache.hadoop.hive.cli.CliDriver.processFile(CliDriver.java:428) at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:717) at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:684) at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:624) at org.apache.oozie.action.hadoop.HiveMain.runHive(HiveMain.java:335) at org.apache.oozie.action.hadoop.HiveMain.run(HiveMain.java:312) at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:51) at org.apache.oozie.action.hadoop.HiveMain.main(HiveMain.java:69) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:242) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
This is because whenever Atlas is installed, the Hive Hook (which is installed via the Oozie RPM too, in /usr/$stack/$version/atlas/hook/hive/) needs to be copied to the Oozie Share lib in HDFS (e.g., hdfs:///user/oozie/share/lib/lib_$timestamp/hive)
Further, need to save atlas-application.properties used for Hive in /etc/oozie/conf/action-conf/hive
Lastly, remove the atlas.rest.address property from hive-site.xml
Attachments
Attachments
Issue Links
- links to