Uploaded image for project: 'Ambari'
  1. Ambari
  2. AMBARI-18368

Atlas web UI alert after performing stack upgrade to HDP 2.5 and adding Atlas Service

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Critical
    • Resolution: Fixed
    • 2.4.0
    • trunk, 2.4.2
    • stacks
    • None

    Description

      Steps to Reproduce:

      • Install Ambari 2.2.2 with HDP 2.4 and HBase, Kafka, and Hive (this is very important)
      • Kerberize the cluster
      • Perform EU/RU to HDP 2.5
      • Add Atlas Service

      Atlas Server log contains,

      Caused by: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://natu146-ehbs-dgm10toeriesec-u14-1.openstacklocal:8886/solr: Can not find the specified config set: vertex_index
              at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:577)
              at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:241)
              at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:230)
              at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:372)
              at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:325)
              at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1100)
              at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:870)
              at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:806)
              at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:149)
              at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:166)
              at com.thinkaurelius.titan.diskstorage.solr.Solr5Index.createCollectionIfNotExists(Solr5Index.java:901)
              at com.thinkaurelius.titan.diskstorage.solr.Solr5Index.register(Solr5Index.java:269)
              at com.thinkaurelius.titan.diskstorage.indexing.IndexTransaction.register(IndexTransaction.java:83)
              at com.thinkaurelius.titan.graphdb.database.IndexSerializer.register(IndexSerializer.java:92)
              at com.thinkaurelius.titan.graphdb.database.management.ManagementSystem.addIndexKey(ManagementSystem.java:534)
              at org.apache.atlas.repository.graph.GraphBackedSearchIndexer.enhanceMixedIndex(GraphBackedSearchIndexer.java:405)
              at org.apache.atlas.repository.graph.GraphBackedSearchIndexer.createIndexes(GraphBackedSearchIndexer.java:334)
              at org.apache.atlas.repository.graph.GraphBackedSearchIndexer.initialize(GraphBackedSearchIndexer.java:103)
              ... 71 more
      

      Atlas tables in HBase look ok.

      su hbase
      kinit -kt /etc/security/keytabs/hbase.headless.keytab cstm-hbase@EXAMPLE.COM
      hbase shell
      
      hbase(main):001:0> list
      TABLE
      ATLAS_ENTITY_AUDIT_EVENTS
      atlas_titan
      2 row(s) in 1.4300 seconds
      
      => ["ATLAS_ENTITY_AUDIT_EVENTS", "atlas_titan"]
      

      Workaround

      1. Stop Atlas Server
      2. Copy solr xml files to correct config folder and chown as $atlas_user:$hadoop_group

      cp -R /usr/hdp/2.5.0.0-####/etc/atlas/conf.dist/solr/* /etc/atlas/conf/solr/
      cp: overwrite `/etc/atlas/conf/solr/solrconfig.xml'? n
      chown atlas:hadoop /etc/atlas/conf/solr/*
      
      cp /usr/hdp/2.5.0.0-####/etc/atlas/conf.dist/users-credentials.properties /etc/atlas/conf/
      cp /usr/hdp/2.5.0.0-####/etc/atlas/conf.dist/policy-store.txt /etc/atlas/conf/
      
      chown atlas:hadoop /etc/atlas/conf/users-credentials.properties
      chown atlas:hadoop /etc/atlas/conf/policy-store.txt
      

      3. Delete zookeeper znode,

      # kinit -kt /etc/security/keytabs/atlas.service.keytab  atlas/<HOST>@<DOMAIN>
      # cd /usr/hdp/current/zookeeper-client/bin/ 
      # ./zkCli.sh -server <zookeepernode>:<zookeeperport>
      [ ...... (CONNECTED) ] rmr  /infra-solr/configs/atlas_configs
      

      4. Ensure Atlas application-properties are present for,
      atlas.jaas.KafkaClient.option.keyTab = /etc/security/keytabs/atlas.service.keytab
      atlas.jaas.KafkaClient.option.principal = atlas/_HOST@EXAMPLE.COM

      5. Start Atlas

      Attachments

        1. AMBARI-18368.patch
          2 kB
          Alejandro Fernandez
        2. AMBARI-18368.addendum.patch
          6 kB
          Alejandro Fernandez

        Issue Links

          Activity

            People

              afernandez Alejandro Fernandez
              afernandez Alejandro Fernandez
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: