Uploaded image for project: 'Ambari'
  1. Ambari
  2. AMBARI-18038

HBase start failing with API

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Critical
    • Resolution: Fixed
    • 2.4.0
    • 2.4.0
    • ambari-server

    Description

      UrlPath : /api/v1/clusters/cl1
      Using username : admin
      Request body : {"RequestInfo":

      {"command":"RESTART","context":"Restart all required services","operation_level":"host_component"}

      ,"Requests/resource_filters":[

      {"hosts_predicate":"HostRoles/stale_configs=true"}

      ]}

      Failed task :

      "Traceback (most recent call last):\n  File \"/var/lib/ambari-agent/cache/common-services/HBASE/0.96.0.2.0/package/scripts/hbase_regionserver.py\", line 194, in <module>\n    HbaseRegionServer().execute()\n  File \"/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py\", line 280, in execute\n    method(env)\n  File \"/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py\", line 720, in restart\n    self.start(env, upgrade_type=upgrade_type)\n  File \"/var/lib/ambari-agent/cache/common-services/HBASE/0.96.0.2.0/package/scripts/hbase_regionserver.py\", line 120, in start\n    self.post_start(env, upgrade_type=upgrade_type)\n  File \"/var/lib/ambari-agent/cache/common-services/HBASE/0.96.0.2.0/package/scripts/hbase_regionserver.py\", line 89, in post_start\n    self.apply_atlas_acl(params.hbase_user)\n  File \"/var/lib/ambari-agent/cache/common-services/HBASE/0.96.0.2.0/package/scripts/hbase_regionserver.py\", line 110, in apply_atlas_acl\n    shell.checked_call(format(\"{kinit_cmd}; {permissions_cmd}\"), tries=10, try_sleep=10)\n  File \"/usr/lib/python2.6/site-packages/resource_management/core/shell.py\", line 71, in inner\n    result = function(command, **kwargs)\n  File \"/usr/lib/python2.6/site-packages/resource_management/core/shell.py\", line 93, in checked_call\n    tries=tries, try_sleep=try_sleep)\n  File \"/usr/lib/python2.6/site-packages/resource_management/core/shell.py\", line 141, in _call_wrapper\n    result = _call(command, **kwargs_copy)\n  File \"/usr/lib/python2.6/site-packages/resource_management/core/shell.py\", line 294, in _call\n    raise Fail(err_msg)\nresource_management.core.exceptions.Fail: Execution of '/usr/bin/kinit -kt /etc/security/keytabs/hbase.service.keytab hbase/nat-r6-ohis-ambari-api-rbac-6-2.openstacklocal@EXAMPLE.COM; echo \"grant 'atlas', 'RWXCA'\" | hbase shell -n' returned 1. kinit: Permission denied while getting initial credentials\n2016-08-05 04:29:14,573 FATAL [main] conf.Configuration: error parsing conf core-site.xml\njava.io.FileNotFoundException: /etc/hbase/2.5.0.0-1145/0/core-site.xml (Permission denied)\n\tat java.io.FileInputStream.open0(Native Method)\n\tat java.io.FileInputStream.open(FileInputStream.java:195)\n\tat java.io.FileInputStream.<init>(FileInputStream.java:138)\n\tat java.io.FileInputStream.<init>(FileInputStream.java:93)\n\tat sun.net.www.protocol.file.FileURLConnection.connect(FileURLConnection.java:90)\n\tat sun.net.www.protocol.file.FileURLConnection.getInputStream(FileURLConnection.java:188)\n\tat java.net.URL.openStream(URL.java:1045)\n\tat org.apache.hadoop.conf.Configuration.parse(Configuration.java:2502)\n\tat org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2573)\n\tat org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2526)\n\tat org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2418)\n\tat org.apache.hadoop.conf.Configuration.set(Configuration.java:1143)\n\tat org.apache.hadoop.conf.Configuration.set(Configuration.java:1115)\n\tat org.apache.hadoop.conf.Configuration.setBoolean(Configuration.java:1451)\n\tat org.apache.hadoop.hbase.io.compress.Compression$Algorithm.<init>(Compression.java:249)\n\tat org.apache.hadoop.hbase.io.compress.Compression$Algorithm.<init>(Compression.java:105)\n\tat org.apache.hadoop.hbase.io.compress.Compression$Algorithm$1.<init>(Compression.java:106)\n\tat org.apache.hadoop.hbase.io.compress.Compression$Algorithm.<clinit>(Compression.java:106)\n\tat org.apache.hadoop.hbase.HColumnDescriptor.<clinit>(HColumnDescriptor.java:135)\n\tat java.lang.Class.forName0(Native Method)\n\tat java.lang.Class.forName(Class.java:348)\n\tat org.jruby.javasupport.JavaSupport.loadJavaClass(JavaSupport.java:136)\n\tat org.jruby.javasupport.JavaSupport.loadJavaClassVerbose(JavaSupport.java:145)\n\tat org.jruby.javasupport.JavaClass.forNameVerbose(JavaClass.java:1193)\n\tat org.jruby.javasupport.Java.getProxyOrPackageUnderPackage(Java.java:914)\n\tat org.jruby.javasupport.Java.get_proxy_or_package_under_package(Java.java:947)\n\tat org.jruby.javasupport.JavaUtilities.get_proxy_or_package_under_package(JavaUtilities.java:54)\n\tat org.jruby.javasupport.JavaUtilities$s$2$0$get_proxy_or_package_under_package.call(JavaUtilities$s$2$0$get_proxy_or_package_under_package.gen:65535)\n\tat org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:201)\n\tat org.jruby.ast.CallTwoArgNode.interpret(CallTwoArgNode.java:59)\n\tat org.jruby.ast.NewlineNode.interpret(NewlineNode.java:104)\n\tat org.jruby.ast.BlockNode.interpret(BlockNode.java:71)\n\tat org.jruby.evaluator.ASTInterpreter.INTERPRET_METHOD(ASTInterpreter.java:74)\n\tat org.jruby.internal.runtime.methods.InterpretedMethod.call(InterpretedMethod.java:120)\n\tat org.jruby.internal.runtime.methods.DefaultMethod.call(DefaultMethod.java:165)\n\tat org.jruby.javasupport.util.RuntimeHelpers$MethodMissingMethod.call(RuntimeHelpers.java:497)\n\tat org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:203)\n\tat org.jruby.runtime.callsite.CachingCallSite.callMethodMissing(CachingCallSite.java:379)\n\tat org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:289)\n\tat org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:135)\n\tat org.jruby.ast.CallNoArgNode.interpret(CallNoArgNode.java:63)\n\tat org.jruby.ast.CallNoArgNode.interpret(CallNoArgNode.java:63)\n\tat org.jruby.ast.FCallOneArgNode.interpret(FCallOneArgNode.java:36)\n\tat org.jruby.ast.NewlineNode.interpret(NewlineNode.java:104)\n\tat org.jruby.ast.BlockNode.interpret(BlockNode.java:71)\n\tat org.jruby.evaluator.ASTInterpreter.INTERPRET_CLASS(ASTInterpreter.java:103)\n\tat org.jruby.evaluator.ASTInterpreter.evalClassDefinitionBody(ASTInterpreter.java:260)\n\tat org.jruby.ast.ModuleNode.interpret(ModuleNode.java:120)\n\tat org.jruby.ast.NewlineNode.interpret(NewlineNode.java:104)\n\tat org.jruby.ast.BlockNode.interpret(BlockNode.java:71)\n\tat org.jruby.ast.RootNode.interpret(RootNode.java:129)\n\tat org.jruby.evaluator.ASTInterpreter.INTERPRET_ROOT(ASTInterpreter.java:119)\n\tat org.jruby.Ruby.runInterpreter(Ruby.java:724)\n\tat org.jruby.Ruby.loadFile(Ruby.java:2489)\n\tat org.jruby.runtime.load.ExternalScript.load(ExternalScript.java:66)\n\tat org.jruby.runtime.load.LoadService.tryLoadingLibraryOrScript(LoadService.java:751)\n\tat org.jruby.runtime.load.LoadService.smartLoad(LoadService.java:332)\n\tat org.jruby.runtime.load.LoadService.require(LoadService.java:381)\n\tat org.jruby.runtime.load.LoadService.lockAndRequire(LoadService.java:304)\n\tat org.jruby.RubyKernel.requireCommon(RubyKernel.java:1079)\n\tat org.jruby.RubyKernel.require(RubyKernel.java:1062)\n\tat org.jruby.RubyKernel$s$1$0$require.call(RubyKernel$s$1$0$require.gen:65535)\n\tat org.jruby.internal.runtime.methods.JavaMethod$JavaMethodOneOrNBlock.call(JavaMethod.java:319)\n\tat org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:312)\n\tat org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:169)\n\tat usr.hdp.$2_dot_5_dot_0_dot_0_minus_1145.hbase.bin.hirb.__file__(/usr/hdp/2.5.0.0-1145/hbase/bin/hirb.rb:118)\n\tat usr.hdp.$2_dot_5_dot_0_dot_0_minus_1145.hbase.bin.hirb.load(/usr/hdp/2.5.0.0-1145/hbase/bin/hirb.rb)\n\tat org.jruby.Ruby.runScript(Ruby.java:697)\n\tat org.jruby.Ruby.runScript(Ruby.java:690)\n\tat org.jruby.Ruby.runNormally(Ruby.java:597)\n\tat org.jruby.Ruby.runFromMain(Ruby.java:446)\n\tat org.jruby.Main.doRunFromMain(Main.java:369)\n\tat org.jruby.Main.internalRun(Main.java:258)\n\tat org.jruby.Main.run(Main.java:224)\n\tat org.jruby.Main.run(Main.java:208)\n\tat org.jruby.Main.main(Main.java:188)\nNameError: cannot initialize Java class org.apache.hadoop.hbase.HColumnDescriptor\n  get_proxy_or_package_under_package at org/jruby/javasupport/JavaUtilities.java:54\n                      method_missing at file:/grid/0/hdp/2.5.0.0-1145/hbase/lib/jruby-complete-1.6.8.jar!/builtin/javasupport/java.rb:51\n                      HBaseConstants at /usr/hdp/2.5.0.0-1145/hbase/lib/ruby/hbase.rb:91\n                              (root) at /usr/hdp/2.5.0.0-1145/hbase/lib/ruby/hbase.rb:34\n                             require at org/jruby/RubyKernel.java:1062\n                              (root) at /usr/hdp/2.5.0.0-1145/hbase/bin/hirb.rb:118\n2016-08-05 04:29:14,697 FATAL [Thread-1] conf.Configuration: error parsing conf core-site.xml\njava.io.FileNotFoundException: /etc/hbase/2.5.0.0-1145/0/core-site.xml (Permission denied)\n\tat java.io.FileInputStream.open0(Native Method)\n\tat java.io.FileInputStream.open(FileInputStream.java:195)\n\tat java.io.FileInputStream.<init>(FileInputStream.java:138)\n\tat java.io.FileInputStream.<init>(FileInputStream.java:93)\n\tat sun.net.www.protocol.file.FileURLConnection.connect(FileURLConnection.java:90)\n\tat sun.net.www.protocol.file.FileURLConnection.getInputStream(FileURLConnection.java:188)\n\tat java.net.URL.openStream(URL.java:1045)\n\tat org.apache.hadoop.conf.Configuration.parse(Configuration.java:2502)\n\tat org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2573)\n\tat org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2526)\n\tat org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2418)\n\tat org.apache.hadoop.conf.Configuration.get(Configuration.java:981)\n\tat org.apache.hadoop.conf.Configuration.getTrimmed(Configuration.java:1031)\n\tat org.apache.hadoop.conf.Configuration.getBoolean(Configuration.java:1432)\n\tat org.apache.hadoop.hbase.HBaseConfiguration.checkDefaultsVersion(HBaseConfiguration.java:68)\n\tat org.apache.hadoop.hbase.HBaseConfiguration.addHbaseResources(HBaseConfiguration.java:82)\n\tat org.apache.hadoop.hbase.HBaseConfiguration.create(HBaseConfiguration.java:97)\n\tat org.apache.phoenix.query.ConfigurationFactory$ConfigurationFactoryImpl$1.call(ConfigurationFactory.java:49)\n\tat org.apache.phoenix.query.ConfigurationFactory$ConfigurationFactoryImpl$1.call(ConfigurationFactory.java:46)\n\tat org.apache.phoenix.util.PhoenixContextExecutor.call(PhoenixContextExecutor.java:78)\n\tat org.apache.phoenix.util.PhoenixContextExecutor.callWithoutPropagation(PhoenixContextExecutor.java:93)\n\tat org.apache.phoenix.query.ConfigurationFactory$ConfigurationFactoryImpl.getConfiguration(ConfigurationFactory.java:46)\n\tat org.apache.phoenix.jdbc.PhoenixDriver$1.run(PhoenixDriver.java:84)\nException in thread \"Thread-1\" java.lang.RuntimeException: java.io.FileNotFoundException: /etc/hbase/2.5.0.0-1145/0/core-site.xml (Permission denied)\n\tat org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2673)\n\tat org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2526)\n\tat org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2418)\n\tat org.apache.hadoop.conf.Configuration.get(Configuration.java:981)\n\tat org.apache.hadoop.conf.Configuration.getTrimmed(Configuration.java:1031)\n\tat org.apache.hadoop.conf.Configuration.getBoolean(Configuration.java:1432)\n\tat org.apache.hadoop.hbase.HBaseConfiguration.checkDefaultsVersion(HBaseConfiguration.java:68)\n\tat org.apache.hadoop.hbase.HBaseConfiguration.addHbaseResources(HBaseConfiguration.java:82)\n\tat org.apache.hadoop.hbase.HBaseConfiguration.create(HBaseConfiguration.java:97)\n\tat org.apache.phoenix.query.ConfigurationFactory$ConfigurationFactoryImpl$1.call(ConfigurationFactory.java:49)\n\tat org.apache.phoenix.query.ConfigurationFactory$ConfigurationFactoryImpl$1.call(ConfigurationFactory.java:46)\n\tat org.apache.phoenix.util.PhoenixContextExecutor.call(PhoenixContextExecutor.java:78)\n\tat org.apache.phoenix.util.PhoenixContextExecutor.callWithoutPropagation(PhoenixContextExecutor.java:93)\n\tat org.apache.phoenix.query.ConfigurationFactory$ConfigurationFactoryImpl.getConfiguration(ConfigurationFactory.java:46)\n\tat org.apache.phoenix.jdbc.PhoenixDriver$1.run(PhoenixDriver.java:84)\nCaused by: java.io.FileNotFoundException: /etc/hbase/2.5.0.0-1145/0/core-site.xml (Permission denied)\n\tat java.io.FileInputStream.open0(Native Method)\n\tat java.io.FileInputStream.open(FileInputStream.java:195)\n\tat java.io.FileInputStream.<init>(FileInputStream.java:138)\n\tat java.io.FileInputStream.<init>(FileInputStream.java:93)\n\tat sun.net.www.protocol.file.FileURLConnection.connect(FileURLConnection.java:90)\n\tat sun.net.www.protocol.file.FileURLConnection.getInputStream(FileURLConnection.java:188)\n\tat java.net.URL.openStream(URL.java:1045)\n\tat org.apache.hadoop.conf.Configuration.parse(Configuration.java:2502)\n\tat org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2573)\n\t... 14 more"
      

      Attachments

        1. rb50849.patch
          1 kB
          Sumit Mohanty

        Issue Links

          Activity

            People

              dmitriusan Dmitry Lysnichenko
              shreyabhatm@gmail.com Shreya Bhat
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: