Uploaded image for project: 'Ambari'
  1. Ambari
  2. AMBARI-22522

Livy server fails to start during downgrade due to absence of 'conf' directory

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Blocker
    • Resolution: Fixed
    • 2.6.1
    • 2.6.1
    • ambari-server
    • None

    Description

      STR

      1. Deployed cluster with Ambari version: 2.4.3.0-30 and HDP version: 2.5.5.0-157
      2. Upgrade Ambari to 2.6.1.0-43
      3. Start EU to 2.6.4.0-36 and downgrade at final step

      Result
      Observed during downgrade that Livy server start failed with below error:

      Traceback (most recent call last):
        File "/var/lib/ambari-agent/cache/common-services/SPARK/1.2.1/package/scripts/livy_server.py", line 141, in <module>
          LivyServer().execute()
        File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 367, in execute
          method(env)
        File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 970, in restart
          self.start(env, upgrade_type=upgrade_type)
        File "/var/lib/ambari-agent/cache/common-services/SPARK/1.2.1/package/scripts/livy_server.py", line 61, in start
          self.configure(env)
        File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 120, in locking_configure
          original_configure(obj, *args, **kw)
        File "/var/lib/ambari-agent/cache/common-services/SPARK/1.2.1/package/scripts/livy_server.py", line 51, in configure
          setup_livy(env, 'server', upgrade_type=upgrade_type, action = 'config')
        File "/var/lib/ambari-agent/cache/common-services/SPARK/1.2.1/package/scripts/setup_livy.py", line 56, in setup_livy
          mode=0644,
        File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 166, in __init__
          self.env.run()
        File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 160, in run
          self.run_action(resource, action)
        File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 124, in run_action
          provider_action()
        File "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py", line 120, in action_create
          raise Fail("Applying %s failed, parent directory %s doesn't exist" % (self.resource, dirname))
      resource_management.core.exceptions.Fail: Applying File['/usr/hdp/current/livy-server/conf/livy-env.sh'] failed, parent directory /usr/hdp/current/livy-server/conf doesn't exist
      

      On livy node, observed the following:

      root@ctr-e134-1499953498516-324773-01-000007:/usr/hdp/current/livy-server# ls -la /usr/hdp/current/livy-server/
      total 40
      drwxr-xr-x  8 root root  4096 Nov 20 23:49 .
      drwxr-xr-x 42 root root  4096 Nov 21 00:05 ..
      drwxr-xr-x  2 root root  4096 Nov 20 23:43 bin
      lrwxrwxrwx  1 root root    14 Apr 21  2017 conf -> /etc/livy/conf
      drwxr-xr-x  3 root root  4096 Nov 20 23:43 etc
      drwxr-xr-x  2 root root 12288 Nov 20 23:43 jars
      drwxr-xr-x  2 livy livy  4096 Nov 20 23:49 logs
      drwxr-xr-x  2 root root  4096 Nov 20 23:43 repl-jars
      drwxr-xr-x  2 root root  4096 Nov 20 23:43 rsc-jars
      
      ===============
      root@ctr-e134-1499953498516-324773-01-000007:~# ls -la /etc/livy/
      total 16
      drwxr-xr-x   4 root root 4096 Nov 21 04:32 .
      drwxr-xr-x 120 root root 4096 Nov 21 03:26 ..
      drwxr-xr-x   3 root root 4096 Nov 21 03:30 2.6.4.0-36
      lrwxrwxrwx   1 root root   21 Nov 21 04:32 conf -> /usr/hdp/current/livy
      drwxr-xr-x   2 root root 4096 Nov 20 23:49 conf.backup
      

      Looks like there is no directory '/etc/livy/2.5.5.0-157/0'

      From downgrade wizard, found that 'Set Version on All Hosts' Upgrade group printed the following for livy:

      2017-11-21 04:32:08,022 - Link['/etc/livy/conf'] {'to': '/usr/hdp/current/livy'}
      2017-11-21 04:32:08,022 - Link['/etc/livy/conf'] replacing old symlink to /grid/0/hdp/current/livy
      2017-11-21 04:32:08,022 - Warning: linking to nonexistent location /usr/hdp/current/livy
      2017-11-21 04:32:08,022 - Creating symbolic Link['/etc/livy/conf'] to /usr/hdp/current/livy
      

      The cause of this appears to be that livy doesn't create a conf directory as expected:

      Skipping the conf-select tool on livy since /etc/livy/conf does not exist.
      

      Attachments

        1. AMBARI-22522.patch
          1 kB
          Jonathan Hurley

        Issue Links

          Activity

            People

              jonathanhurley Jonathan Hurley
              jonathanhurley Jonathan Hurley
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: