Uploaded image for project: 'Bigtop'
  1. Bigtop
  2. BIGTOP-3695

Define Bigtop 3.2 release BOM

    XMLWordPrintableJSON

Details

    • Task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • None
    • None
    • None

    Description

      Let's discuss the component stack of the next release.
      I'd like to propose upgrading Hadoop to 3.3.x since we have put it off long time.
      I'd also like to add Ranger to our stack, which is often required in the projects with high security requirements.

      Components

      Components                   in v3.1      in v3.2
alluxio                        2.4.1  =>  2.8.0 [1]
bigtop-ambari-mpack            2.7.5      2.7.5
bigtop-groovy                  2.5.4      2.5.4
bigtop-jsvc                    1.2.4      1.2.4
bigtop-utils                   3.1.0      3.1.0
elasticsearch                 5.6.14  =>  dropped
flink                         1.11.6  =>  1.15.3
gpdb                          5.28.5      5.28.5
hadoop                         3.2.3  =>  3.3.4
hbase                         2.4.11  =>  2.4.13 [2]
hive                           3.1.2  =>  3.1.3 [3]
kafka                          2.8.1      2.8.1
kibana                         5.4.1  =>  dropped
livy                           0.7.1      0.7.1
logstash                       5.4.1  =>  dropped
oozie                          5.2.1      5.2.1
phoenix                        5.1.2      5.1.2
ranger                           n/a  =>  2.2.0 [4]
solr                          8.11.1  =>  8.11.2
spark                          3.1.2  =>  3.2.1 [5]
sqoop                          1.4.7      dropped
tez                           0.10.1      0.10.1
ycsb                          0.17.0      0.17.0
zeppelin                      0.10.0  =>  0.10.1 [6]
zookeeper                      3.5.9      3.5.9

      Distros

      - CentOS 7, Rocky Linux 8
- Debian 10, 11
- Fedora 36
- Ubuntu 22.04, 20.04

      Archs

      - x86_64
- aarch64
- ppc64le

      JDK

      JDK8

      [1]: According to the following document, Alluxio 2.8.0 is supposed to be compatible with HDFS 3.3.x.
      https://docs.alluxio.io/os/user/2.8.0/en/ufs/HDFS.html#supported-hdfs-versions

      [2]: According to the following document, HBase 2.4.12 is supposed to be compatible with Hadoop 3.3.x.
      https://hbase.apache.org/book.html#hadoop

      [3]: According to the following document, Hive 3.1.3 is supposed to be compatible with Hadoop 3.x.y.
      https://hive.apache.org/downloads.html

      [4]: Ranger 2.2.0 is built with Hadoop 3.3.0 by default, so we can expect it's compatible with Hadoop 3.3.3.
      https://github.com/apache/ranger/blob/release-ranger-2.2.0/pom.xml#L121

      [5]: We can expect Spark 3.2.1 is compatible with Hadoop 3.3.x, since it's built for Hadoop 3.3.0 in the following example:
      https://spark.apache.org/docs/3.2.1/building-spark.html#specifying-the-hadoop-version-and-enabling-yarn

      [6]: Zeppelin 0.10.1, which is the latest release as of now, already has the "spark-3.2" build profile (https://github.com/apache/zeppelin/blob/v0.10.1/spark/pom.xml#L172), but doesn't the Flink shim for 1.15.x yet (https://github.com/apache/zeppelin/tree/v0.10.1/flink).
      Probably we have to apply ZEPPELIN-5600.

      Attachments

        Issue Links

          contains

          Sub-task - The sub-task of the issue BIGTOP-3786 Improve the build scripts to append suffix to the rpm package name

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. BIGTOP-3749 Bug-fixes for Bigtop-3.2 release

          • Major - Major loss of function.
          • Resolved

          New Feature - A new feature of the product, which has yet to be developed. BIGTOP-3403 Support Hadoop 3.3 for Bigtop

          • Major - Major loss of function.
          • Resolved

          New Feature - A new feature of the product, which has yet to be developed. BIGTOP-3830 Introduce bigtop-select package to Bigtop

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. BIGTOP-3689 Upgrade Mpack services stack to Bigtop 3.2

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. BIGTOP-3748 Upgrade Bigtop components for 3.2-release

          • Major - Major loss of function.
          • Resolved

          Task - A task that needs to be done. BIGTOP-3759 Remove ELK from Bigtop 3.2.0 stack

          • Major - Major loss of function.
          • Resolved
          Dependency

          Improvement - An improvement or enhancement to an existing feature or task. BIGTOP-3784 Improve the build scripts to make binary distribution more configurable

          • Major - Major loss of function.
          • Open

          Activity

            People

              yqGu Yuqi Gu
              sekikn Kengo Seki
              Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 10m
                  10m