Uploaded image for project: 'Ranger'
  1. Ranger
  2. RANGER-4309

Add prerequisites with minimal plugin versions for the regular build process

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Open
    • Minor
    • Resolution: Unresolved
    • 2.4.0
    • None
    • documentation, plugins
    • None
    • CentOS 7.9 with JDK-8 and JDK-11, Apache Maven 3.9.3

    Description

      To start simplifying the work of people who want to build the project using the standard build process, we would like to propose an update of the README.md with the minimum versions necessary for the building of Apache ranger with the plugins and dependencies on them.

      For the moment, 4 prerequisites would be indicated:

      • JDK 1.8+ taken from pom.xml
      • Apache Maven 3.6.3+ taken from pom.xml
      • Hadoop 3.3.0+ taken from tests
      • Hive 3.0.0+ taken from tests

      Hadoop 3.3.0+

      The method CheckPermissionWithContext was not implemented before hadoop 3.3.0 and it is used during the building process.
      Here is the file in which this method is implemented since 3.3.0 and it's possible to see that it wasn't at the release 3.2.4 :
      https://github.com/apache/hadoop/blob/release-3.2.4-RC0/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeAttributeProvider.java

      It is necessary to build the latest version, if we try to build with the hadoop.version flag pointing to 3.2.4 version we get the following error:

      [ERROR] /usr/hdp/ranger/hdfs-agent/src/main/java/org/apache/ranger/authorization/hadoop/RangerHdfsAuthorizer.java:[229,2] error: method does not override or implement a method from a supertype

      Hive 3.0.0+

      At the beginning, we wanted to build Apache Ranger with hive.version=2.3.3.

      [ERROR] Failed to execute goal on project ranger-hive-plugin: Could not resolve dependencies for project org.apache.ranger:ranger-hive-plugin:jar:3.0.0-SNAPSHOT: Failed to collect dependencies at org.apache.hive:hive-service:jar:2.3.3 -> org.apache.hive:hive-exec:jar:2.3.3 -> org.apache.calcite:calcite-core:jar:1.10.0 -> org.pentaho:pentaho-aggdesigner-algorithm:jar:5.1.5-jhyde

      The approach we have tried to fix this was to add the following to both ranger-hive-plugin-shim/pom.xml and hive-agent/pom.xml for both hive-exec and hive-service artifacts inside <exclusions></exclusions>:

      <exclusion> 
          <groupId>org.pentaho</groupId> 
          <artifactId>*</artifactId> 
      </exclusion>

      We got the following error (for example with hive.version=2.8.1):

      [ERROR] Failed to execute goal on project ranger-hive-plugin: Could not resolve dependencies for project org.apache.ranger:ranger-hive-plugin:jar:3.0.0-SNAPSHOT: The following artifacts could not be resolved: org.apache.hive:hive-common:jar:2.8.1 (absent), org.apache.hive:hive-service:jar:2.8.1 (absent), org.apache.hive:hive-exec:jar:2.8.1 (absent), org.apache.hive:hive-metastore:jar:2.8.1 (absent), org.apache.hive:hive-jdbc:jar:2.8.1 (absent): org.apache.hive:hive-common:jar:2.8.1 was not found in https://repository.apache.org/content/repositories/snapshots during a previous attempt. This failure was cached in the local repository and resolution is not reattempted until the update interval of apache.snapshots.https has elapsed or updates are forced

      We had the same errors for all versions < 3.0.0

       

      Tested by clement.lucas@cern.ch and emil.kleszcz@cern.ch.

      Attachments

        Activity

          People

            sulray Clément Lucas
            sulray Clément Lucas
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:

              Time Tracking

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0h
                0h
                Logged:
                Time Spent - 10m
                10m