Uploaded image for project: 'Bigtop'
  1. Bigtop
  2. BIGTOP-3471

Define Bigtop 3.0 release BOM

    XMLWordPrintableJSON

Details

    • Task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 3.0.0
    • None
    • None

    Description

      It's about time to discuss the next release BOM

      Based on the past discussions [1][2], we're going to focus "core" components in the next release and drop the ones (at least temporarily) which are

      • not so widely used by our users
      • not actively developed recently
      • incompatible (or not sure if compatible and no enough time/resource to investigate it) with Hadoop 3.x

      so that we can publish our releases more frequently with limited resources.

      To drop such components, I marked the ones that matches any of the above conditions as "to be removed" in the below list.
      (Tez, Phoenix and Livy seem to be inactive recently, but they significantly improve performance or usability of Hadoop/Hive, HBase and Spark respectively, so I'd like to keep them as an exception, from my biased perspective)
      So please update the BOM if your required components are going to be removed!

      Components

      Components               in v1.5    in v3.0
      alluxio                   1.8.2  => 2.4.1
      ambari                    2.6.1  => 2.7.5
      bigtop-ambari-mpack       2.6.1  => 2.7.5
      bigtop_groovy             2.5.4  => 2.5.4
      bigtop_jsvc               1.0.15 => 1.0.15
      bigtop_tomcat             8.5.57 => removed
      bigtop_utils              1.5.0  => 3.0.0
      elasticsearch             5.6.14 => 5.6.14 (or greater, as far as its license is ASLv2-compliant)
      flink                     1.6.4  => 1.11.3 [3]
      flume                     1.9.0  => removed
      giraph                    1.2.0  => removed
      gpdb                      5.10.0 => 5.28.5
      hadoop                    2.10.1 => 3.2.1 (or 3.3.x (BIGTOP-3403)? see also the discussions on BIGTOP-3280 and [4])
      hbase                     1.5.0  => 2.2.6 (or 2.3.3 [5], if possible)
      hive                      2.3.6  => 3.1.2 [6]
      ignite_hadoop             2.7.6  => removed
      kafka                     2.4.0  => 2.4.1
      kibana                    5.4.1  => 5.4.1 (or greater, as far as its license is ASLv2-compliant)
      livy                      0.7.0  => 0.7.1
      logstash                  5.4.1  => 5.4.1 (or greater, as far as its license is ASLv2-compliant)
      mahout                    0.13.0 => removed
      oozie                     4.3.0  => 5.2.1
      phoenix                   4.15.0-HBase-1.5 => 5.1.0 [7]
      qfs                       2.0.0  => removed
      solr                      6.6.6  => 8.7.0 (or maybe 8.8.0?)
      spark                     2.4.5  => 3.0.1
      sqoop                     1.4.6  => 1.4.7
      sqoop2                    1.99.4 => removed
      tez                       0.9.2  => 0.10.0
      ycsb                      0.12.0 => 0.17.0
      zeppelin                  0.8.2  => 0.9.0 [8]
      zookeeper                 3.4.13 => 3.4.14 (or 3.5+ [9] if possible)
      

      Distros

      - CentOS 7, 8
      - Debian 9 (except for ppc64le, because its EOL LTS is not provided for ppc64le and its official Docker image also seems to have been removed from DockerHub as of May 2021), 10
      - Fedora 33
      - Ubuntu 18.04, 20.04
      

      Some points of discussion:

      • Should we still support CentOS 8 which is supposed to be EOL'd within 2021 [10]?
      • Should we update Fedora to 33, the latest version as of Jan 2021?
      • Should we also support Ubuntu 20.04 additionally?

      Archs

      - x86_64
      - aarch64
      - ppc64le (if the CI server is recovered)
      

      JDK

      JDK8
      

      Also, I tentatively assigned 3.0 as its version for the following reasons, but let's discuss it too if someone has another option.

      • We're going to adopt Hadoop 3.x (and hopefully Spark 3.x), so Bigtop 3.0 is aligned with them [11].
      • Bigtop 2.x is (unofficially) reserved for the "Cloud Native Bigtop" (BIGTOP-3225) release line [12].
      • In my opinion, this major version leap is good to impress users backward incompatibility described above.

      [1]: https://lists.apache.org/thread.html/fe54be35856c8f6a23fb7d35b979e3dc90ad58baf94d15a206cdaf63%40%3Cdev.bigtop.apache.org%3E
      [2]: https://lists.apache.org/thread.html/r71869111448bf5f8bc82e7031ff25d5455ab957e8e536d000026fe16%40%3Cdev.bigtop.apache.org%3E
      [3]: The latest release which is officially supported by Zeppelin 0.9.0. https://twitter.com/ApacheZeppelin/status/1288855217895698433
      [4]: https://github.com/apache/bigtop/pull/704
      [5]: The latest release which is officially compatible with Hadoop 3.2.x as of now. https://hbase.apache.org/book.html#hadoop
      [6]: The latest release which is officially compatible with Hadoop 3.x.y as of now. https://hive.apache.org/downloads.html#26-august-2019-release-312-available
      [7]: The latest release which is officially compatible with HBase 2.x as of now. https://phoenix.apache.org/download.html
      [8]: The latest release which officially supports Spark 3.0 as of now. https://twitter.com/ApacheZeppelin/status/1288855217895698433
      [9]: HBase 2.3.3 is built with Zookeeper 3.5.7 by default and doesn't seem to be built with Zookeeper 3.4.x as it is.
      [10]: https://blog.centos.org/2020/12/future-is-centos-stream/
      [11]: https://issues.apache.org/jira/browse/BIGTOP-3287?focusedCommentId=17018665&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17018665
      [12]: https://lists.apache.org/thread.html/e0af97d83c33bf370e25fca61aee95d0ed63193893c5f0ff7d187ebb%40%3Cdev.bigtop.apache.org%3E

      Attachments

        1. 1.jpg
          9 kB
          Andrey Valiew
        2. 1.jpg
          9 kB
          Andrey Valiew
        3. 2.jpg
          38 kB
          Andrey Valiew
        4. build_matrix.jpg
          244 kB
          Kengo Seki
        There are no Sub-Tasks for this issue.

        Activity

          People

            yqGu Yuqi Gu
            sekikn Kengo Seki
            Votes:
            5 Vote for this issue
            Watchers:
            20 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Time Tracking

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0h
                0h
                Logged:
                Time Spent - 36h 40m
                36h 40m