Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 2.8.0
    • 2.9.0
    • ambari-server
    • None

    Description

      Add ranger 2.4 support in ambari bigtop stacks

      Currently, the work of adapting Ranger to Bigtop and Ambari has been completed. Due to the large number of issues involved in the adaptation, they are summarized as follows:

      apache ambari related issues

      The main issues related to adapting Ranger in Ambari 2.8 are related to the advisor functionality. Enabling Ranger would trigger the advisor to recommend updates to the component's Ranger-related configurations, thus adapting Ranger requires fixing this part first. Otherwise, after enabling Ranger plugin in Ambari, manual updates to the Ranger-related configurations would still be required.

       
       1.AMBARI-25894: Missing file service_advisor.py in some serivces     (merged)

      https://github.com/apache/ambari/pull/3677 

       

      2.AMBARI-25932: Wrong config file name in spark service advisor  (merged)

      https://issues.apache.org/jira/browse/AMBARI-25932
       

      ambari ranger support related issues:

      Here, we additionally adapted Ambari Infra because the Ambari Ranger service relies on the Infra client to perform Solr-related automation settings for Ranger.
       

      1.ambari infra PR

      1.Add support for Ambari Infra in Ambari 2.8 wait for review  (merged)

      https://issues.apache.org/jira/browse/AMBARI-25933

      2.ambari ranger service support (wait for merge in next version)

      https://issues.apache.org/jira/browse/AMBARI-25929

      3.ambari infra service support wait for review (merged)

      https://github.com/apache/ambari/pull/3696

      apache bigtop  related issues

      To adapt Ranger to Ambari, we need to first build RPM packages related to Ranger using Bigtop. The following are the pull requests required for Bigtop to support Ranger.
       
      1.BIGTOP-3925 ranger support  (merged)

      https://github.com/apache/bigtop/pull/1100

      2.BIGTOP-3923: Add missing jars for Ranger (merged)

      https://github.com/apache/bigtop/pull/1099

      3.BIGTOP-3910: Bigtop-select support Ranger   (merged)

      https://github.com/apache/bigtop/pull/1089

      4.BIGTOP-3950: fix ranger etc conf dir     (in review)

      https://github.com/apache/bigtop/pull/1120

       

      Bigtop support for Ranger requires three PRs:

      1. BIGTOP-3925: This PR mainly addresses issues with Ranger RPM packaging and includes a patch that resolves problems with starting HBase after integrating with Ranger.
      1. BIGTOP-3923: This PR addresses package dependency issues when running "java -cp '/usr/bigtop/current/ranger-usersync/lib/*' org.apache.ranger.credentialapi.buildks create ranger.usersync.policymgr.password -value [PROTECTED] -provider jceks://file/usr/bigtop/current/ranger-usersync/conf/ugsync.jceks" command.
      1. BIGTOP-3910: This PR adds support for Ranger in Bigtop-select.

      ranger related issues:

       

      These are the PRs encountered during the process of adapting Ranger in Bigtop Ambari. All 3 PRs have been made into patches and submitted to the aforementioned Bigtop Ranger support-related PRs. The review of the related issues on the Ranger side is also currently underway.
       
       
      1.fix Kafka2.8 can't restart after enable ranger plugin

      https://issues.apache.org/jira/browse/RANGER-4228 wait for review

      2.addresses the issue of HBase not starting after integrating with Ranger due to class loading order

      https://issues.apache.org/jira/browse/RANGER-4201 wait for review

      3.mainly addresses the missing dependency issue when running the Ranger command "java -cp '/usr/bigtop/current/ranger-usersync/lib/*' org.apache.ranger.credentialapi.buildks create ranger.usersync.policymgr.password -value [PROTECTED] -provider jceks://file/usr/bigtop/current/ranger-usersync/conf/ugsync.jceks".

      https://issues.apache.org/jira/browse/RANGER-3992 wait for review

      manual test:

      before enable kerberos all compoent works smoonthly

      after enable kerberos all compoent works smoonthly

      Attachments

        1. image-2023-05-09-11-08-47-864.png
          87 kB
          Jialiang Cai
        2. image-2023-05-09-11-08-57-472.png
          99 kB
          Jialiang Cai
        3. image-2023-05-09-11-09-14-373.png
          92 kB
          Jialiang Cai
        4. image-2023-05-09-11-09-22-441.png
          72 kB
          Jialiang Cai

        Issue Links

          Activity

            wzhy Wang Zheyuan added a comment - - edited

            The latest release of ranger is 2.4.0. How about adding ranger 2.4.0 instead of ranger 2.3.0?

            wzhy Wang Zheyuan added a comment - - edited The latest release of ranger is 2.4.0. How about adding ranger 2.4.0 instead of ranger 2.3.0?
            jialiang Jialiang Cai added a comment - - edited

            wzhy  Great idea. Version 2.4 fixed a lot of bugs. In fact, we have added support for both Ranger 2.3 and 2.4 simultaneously. Only a few modifications are required when packaging Ranger. However, the default Ranger version in the Bigtop community is 2.3. Currently, there is a discussion about whether to upgrade to version 2.4. Once the discussion is concluded, we will update to support Ranger 2.4 here.

            https://issues.apache.org/jira/browse/BIGTOP-3909

             
            jialiang Jialiang Cai added a comment - - edited wzhy   Great idea. Version 2.4 fixed a lot of bugs. In fact, we have added support for both Ranger 2.3 and 2.4 simultaneously. Only a few modifications are required when packaging Ranger. However, the default Ranger version in the Bigtop community is 2.3. Currently, there is a discussion about whether to upgrade to version 2.4. Once the discussion is concluded, we will update to support Ranger 2.4 here. https://issues.apache.org/jira/browse/BIGTOP-3909  
            jialiang Jialiang Cai added a comment -

            wuzhiguo   houyu leiyao 
            The current progress of Ranger adaptation and all related issues are listed in description.
             

            jialiang Jialiang Cai added a comment - wuzhiguo   houyu leiyao   The current progress of Ranger adaptation and all related issues are listed in description.  
            wzhy Wang Zheyuan added a comment -

            Since 2.1.0, Ranger has supported audit to Elasticsearch. Can we support it in Ambari?

            wzhy Wang Zheyuan added a comment - Since 2.1.0, Ranger has supported audit to Elasticsearch. Can we support it in Ambari?
            jialiang Jialiang Cai added a comment - - edited

            wzhy 
            I didn't see any plans for the community to support Elasticsearch in Ambari, but you can develop your own Elasticsearch mpack to easily install it in an existing Ambari cluster.
             Of course, if you're interested, you can also add support for Elasticsearch in Ambari and submit it to the community.
             

            jialiang Jialiang Cai added a comment - - edited wzhy   I didn't see any plans for the community to support Elasticsearch in Ambari, but you can develop your own Elasticsearch mpack to easily install it in an existing Ambari cluster.  Of course, if you're interested, you can also add support for Elasticsearch in Ambari and submit it to the community.  
            bpatel Bhavik Patel added a comment -

            Let's include Ranger-KMS support as well.

            bpatel Bhavik Patel added a comment - Let's include Ranger-KMS support as well.
            bpatel Bhavik Patel added a comment -

             

            1.fix Kafka2.8 can't restart after enable ranger plugin
            https://issues.apache.org/jira/browse/RANGER-4228 wait for review
            2.addresses the issue of HBase not starting after integrating with Ranger due to class loading order
            https://issues.apache.org/jira/browse/RANGER-4201 wait for review
            3.mainly addresses the missing dependency issue when running the Ranger command "java -cp '/usr/bigtop/current/ranger-usersync/lib/*' org.apache.ranger.credentialapi.buildks create ranger.usersync.policymgr.password -value [PROTECTED] -provider jceks://file/usr/bigtop/current/ranger-usersync/conf/ugsync.jceks".
            https://issues.apache.org/jira/browse/RANGER-3992 wait for review
            

             

            Are you observing above issues as ranger-2.3/ranger2.4 does not support hadoop-3.3.5 version.

            https://issues.apache.org/jira/browse/RANGER-4418 : Ranger Jira is there to support hadoop-3.3.6. Once it's fixed for master branch I will backport it to ranger-2.4 branch so we will not observe above issue. 

             

            bpatel Bhavik Patel added a comment -   1.fix Kafka2.8 can't restart after enable ranger plugin https: //issues.apache.org/jira/browse/RANGER-4228 wait for review 2.addresses the issue of HBase not starting after integrating with Ranger due to class loading order https: //issues.apache.org/jira/browse/RANGER-4201 wait for review 3.mainly addresses the missing dependency issue when running the Ranger command "java -cp '/usr/bigtop/current/ranger-usersync/lib/*' org.apache.ranger.credentialapi.buildks create ranger.usersync.policymgr.password -value [PROTECTED] -provider jceks: //file/usr/bigtop/current/ranger-usersync/conf/ugsync.jceks" . https: //issues.apache.org/jira/browse/RANGER-3992 wait for review   Are you observing above issues as ranger-2.3/ranger2.4 does not support hadoop-3.3.5 version. https://issues.apache.org/jira/browse/RANGER-4418 : Ranger Jira is there to support hadoop-3.3.6. Once it's fixed for master branch I will backport it to ranger-2.4 branch so we will not observe above issue.   
            jialiang Jialiang Cai added a comment -

            bpatel 
            If the compatibility issue with Hadoop 3.6 is resolved, I will close the related issue. After resolution, is there a plan to release a minor version of Ranger? This would make it much more convenient for users.
             
             

            jialiang Jialiang Cai added a comment - bpatel   If the compatibility issue with Hadoop 3.6 is resolved, I will close the related issue. After resolution, is there a plan to release a minor version of Ranger? This would make it much more convenient for users.    
            jialiang Jialiang Cai added a comment -

            bpatel  Ranger-KMS support finished ,already merged to master.

            jialiang Jialiang Cai added a comment - bpatel  Ranger-KMS support finished ,already merged to master.
            vjasani Viraj Jasani added a comment -

            jialiang if you are planning to make this available on 2.8.0, you can also create PR against branch-2.8.

            Thank you!

            vjasani Viraj Jasani added a comment - jialiang if you are planning to make this available on 2.8.0, you can also create PR against branch-2.8. Thank you!
            jialiang Jialiang Cai added a comment -

            vjasani I'm also quite confused, as there currently lacks a release plan and roadmap for Ambari. It's uncertain whether to release the Python 2 version of Ambari 2.8 or the Python 3 version of Ambari master.

            jialiang Jialiang Cai added a comment - vjasani I'm also quite confused, as there currently lacks a release plan and roadmap for Ambari. It's uncertain whether to release the Python 2 version of Ambari 2.8 or the Python 3 version of Ambari master.
            vjasani Viraj Jasani added a comment - - edited

            I see, it makes sense to first clarify on the roadmap. IMO, we should only focus on Python 3. We have also recently upgraded to Python 3 and it is working fine in production, just that our prod branch is still based out of 2.6 release line.

            For this Jira, it's fine to keep it on trunk only. We can start some discussion on the roadmap for moving forward with Ambari 2.9 or 3.0 (from trunk branch) and create release of 3.0.0. WDYT?

            vjasani Viraj Jasani added a comment - - edited I see, it makes sense to first clarify on the roadmap. IMO, we should only focus on Python 3. We have also recently upgraded to Python 3 and it is working fine in production, just that our prod branch is still based out of 2.6 release line. For this Jira, it's fine to keep it on trunk only. We can start some discussion on the roadmap for moving forward with Ambari 2.9 or 3.0 (from trunk branch) and create release of 3.0.0. WDYT?
            jialiang Jialiang Cai added a comment -

            I agree, for future releases, we will maintain only the Python 3 version, starting from the master branch, transitioning to 3.0 and using the bigtop stack. Next, we can list some new features or tasks that need support and add them to the roadmap, then proceed to tackle them one by one until completed.

            jialiang Jialiang Cai added a comment - I agree, for future releases, we will maintain only the Python 3 version, starting from the master branch, transitioning to 3.0 and using the bigtop stack. Next, we can list some new features or tasks that need support and add them to the roadmap, then proceed to tackle them one by one until completed.

            People

              jialiang Jialiang Cai
              jialiang Jialiang Cai
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 6h 10m
                  6h 10m