Uploaded image for project: 'Bigtop'
  1. Bigtop
  2. BIGTOP-3471

Define Bigtop 3.0 release BOM

    XMLWordPrintableJSON

Details

    • Task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 3.0.0
    • None
    • None

    Description

      It's about time to discuss the next release BOM

      Based on the past discussions [1][2], we're going to focus "core" components in the next release and drop the ones (at least temporarily) which are

      • not so widely used by our users
      • not actively developed recently
      • incompatible (or not sure if compatible and no enough time/resource to investigate it) with Hadoop 3.x

      so that we can publish our releases more frequently with limited resources.

      To drop such components, I marked the ones that matches any of the above conditions as "to be removed" in the below list.
      (Tez, Phoenix and Livy seem to be inactive recently, but they significantly improve performance or usability of Hadoop/Hive, HBase and Spark respectively, so I'd like to keep them as an exception, from my biased perspective)
      So please update the BOM if your required components are going to be removed!

      Components

      Components               in v1.5    in v3.0
      alluxio                   1.8.2  => 2.4.1
      ambari                    2.6.1  => 2.7.5
      bigtop-ambari-mpack       2.6.1  => 2.7.5
      bigtop_groovy             2.5.4  => 2.5.4
      bigtop_jsvc               1.0.15 => 1.0.15
      bigtop_tomcat             8.5.57 => removed
      bigtop_utils              1.5.0  => 3.0.0
      elasticsearch             5.6.14 => 5.6.14 (or greater, as far as its license is ASLv2-compliant)
      flink                     1.6.4  => 1.11.3 [3]
      flume                     1.9.0  => removed
      giraph                    1.2.0  => removed
      gpdb                      5.10.0 => 5.28.5
      hadoop                    2.10.1 => 3.2.1 (or 3.3.x (BIGTOP-3403)? see also the discussions on BIGTOP-3280 and [4])
      hbase                     1.5.0  => 2.2.6 (or 2.3.3 [5], if possible)
      hive                      2.3.6  => 3.1.2 [6]
      ignite_hadoop             2.7.6  => removed
      kafka                     2.4.0  => 2.4.1
      kibana                    5.4.1  => 5.4.1 (or greater, as far as its license is ASLv2-compliant)
      livy                      0.7.0  => 0.7.1
      logstash                  5.4.1  => 5.4.1 (or greater, as far as its license is ASLv2-compliant)
      mahout                    0.13.0 => removed
      oozie                     4.3.0  => 5.2.1
      phoenix                   4.15.0-HBase-1.5 => 5.1.0 [7]
      qfs                       2.0.0  => removed
      solr                      6.6.6  => 8.7.0 (or maybe 8.8.0?)
      spark                     2.4.5  => 3.0.1
      sqoop                     1.4.6  => 1.4.7
      sqoop2                    1.99.4 => removed
      tez                       0.9.2  => 0.10.0
      ycsb                      0.12.0 => 0.17.0
      zeppelin                  0.8.2  => 0.9.0 [8]
      zookeeper                 3.4.13 => 3.4.14 (or 3.5+ [9] if possible)
      

      Distros

      - CentOS 7, 8
      - Debian 9 (except for ppc64le, because its EOL LTS is not provided for ppc64le and its official Docker image also seems to have been removed from DockerHub as of May 2021), 10
      - Fedora 33
      - Ubuntu 18.04, 20.04
      

      Some points of discussion:

      • Should we still support CentOS 8 which is supposed to be EOL'd within 2021 [10]?
      • Should we update Fedora to 33, the latest version as of Jan 2021?
      • Should we also support Ubuntu 20.04 additionally?

      Archs

      - x86_64
      - aarch64
      - ppc64le (if the CI server is recovered)
      

      JDK

      JDK8
      

      Also, I tentatively assigned 3.0 as its version for the following reasons, but let's discuss it too if someone has another option.

      • We're going to adopt Hadoop 3.x (and hopefully Spark 3.x), so Bigtop 3.0 is aligned with them [11].
      • Bigtop 2.x is (unofficially) reserved for the "Cloud Native Bigtop" (BIGTOP-3225) release line [12].
      • In my opinion, this major version leap is good to impress users backward incompatibility described above.

      [1]: https://lists.apache.org/thread.html/fe54be35856c8f6a23fb7d35b979e3dc90ad58baf94d15a206cdaf63%40%3Cdev.bigtop.apache.org%3E
      [2]: https://lists.apache.org/thread.html/r71869111448bf5f8bc82e7031ff25d5455ab957e8e536d000026fe16%40%3Cdev.bigtop.apache.org%3E
      [3]: The latest release which is officially supported by Zeppelin 0.9.0. https://twitter.com/ApacheZeppelin/status/1288855217895698433
      [4]: https://github.com/apache/bigtop/pull/704
      [5]: The latest release which is officially compatible with Hadoop 3.2.x as of now. https://hbase.apache.org/book.html#hadoop
      [6]: The latest release which is officially compatible with Hadoop 3.x.y as of now. https://hive.apache.org/downloads.html#26-august-2019-release-312-available
      [7]: The latest release which is officially compatible with HBase 2.x as of now. https://phoenix.apache.org/download.html
      [8]: The latest release which officially supports Spark 3.0 as of now. https://twitter.com/ApacheZeppelin/status/1288855217895698433
      [9]: HBase 2.3.3 is built with Zookeeper 3.5.7 by default and doesn't seem to be built with Zookeeper 3.4.x as it is.
      [10]: https://blog.centos.org/2020/12/future-is-centos-stream/
      [11]: https://issues.apache.org/jira/browse/BIGTOP-3287?focusedCommentId=17018665&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17018665
      [12]: https://lists.apache.org/thread.html/e0af97d83c33bf370e25fca61aee95d0ed63193893c5f0ff7d187ebb%40%3Cdev.bigtop.apache.org%3E

      Attachments

        1. 1.jpg
          9 kB
          Andrey Valiew
        2. 1.jpg
          9 kB
          Andrey Valiew
        3. 2.jpg
          38 kB
          Andrey Valiew
        4. build_matrix.jpg
          244 kB
          Kengo Seki
        1.
        Bump Hadoop to 3.2.1 Sub-task Resolved Masatake Iwasaki

        100%

        Original Estimate - Not Specified Original Estimate - Not Specified
        Time Spent - 2.5h
        2.
        Bump Hadoop to 3.2.2 Sub-task Resolved Masatake Iwasaki

        100%

        Original Estimate - Not Specified Original Estimate - Not Specified
        Time Spent - 2h 10m
        3.
        Bump HBase to 2.2.6 Sub-task Resolved Masatake Iwasaki

        100%

        Original Estimate - Not Specified Original Estimate - Not Specified
        Time Spent - 4.5h
        4.
        Bump ZooKeeper to 3.4.14 Sub-task Resolved Masatake Iwasaki

        100%

        Original Estimate - Not Specified Original Estimate - Not Specified
        Time Spent - 1h
        5.
        Bump Tez to 0.10.0 Sub-task Resolved Kengo Seki

        100%

        Original Estimate - Not Specified Original Estimate - Not Specified
        Time Spent - 0.5h
        6.
        Bump Hive to 3.1.2 Sub-task Resolved Kengo Seki

        100%

        Original Estimate - Not Specified Original Estimate - Not Specified
        Time Spent - 1h 20m
        7.
        Bump Spark to 3.0.1 Sub-task Resolved Kengo Seki

        100%

        Original Estimate - Not Specified Original Estimate - Not Specified
        Time Spent - 40m
        8.
        Bump Alluxio to 2.4.1 Sub-task Resolved Kengo Seki

        100%

        Original Estimate - Not Specified Original Estimate - Not Specified
        Time Spent - 0.5h
        9.
        Upgrade Flink to 1.11.3 Sub-task Resolved Yasuaki Okumura

        100%

        Original Estimate - Not Specified Original Estimate - Not Specified
        Time Spent - 1h
        10.
        Bump Sqoop to 1.4.7 Sub-task Resolved Kengo Seki

        100%

        Original Estimate - Not Specified Original Estimate - Not Specified
        Time Spent - 0.5h
        11.
        Bump Kafka to 2.4.1 Sub-task Resolved Kengo Seki

        100%

        Original Estimate - Not Specified Original Estimate - Not Specified
        Time Spent - 1h
        12.
        Add Ubuntu 20.04 support to the Bigtop toolchain Sub-task Resolved Kengo Seki

        100%

        Original Estimate - Not Specified Original Estimate - Not Specified
        Time Spent - 1h
        13.
        Add Ubuntu 20.04 support to the Docker provisioner Sub-task Resolved Kengo Seki

        100%

        Original Estimate - Not Specified Original Estimate - Not Specified
        Time Spent - 1h
        14.
        Bump YCSB to 0.17.0 Sub-task Resolved Kengo Seki

        100%

        Original Estimate - Not Specified Original Estimate - Not Specified
        Time Spent - 0.5h
        15.
        Bump Phoenix to 5.1.0 Sub-task Resolved Kengo Seki

        100%

        Original Estimate - Not Specified Original Estimate - Not Specified
        Time Spent - 50m
        16.
        Bump Zeppelin to 0.9.0 Sub-task Resolved Kengo Seki

        100%

        Original Estimate - Not Specified Original Estimate - Not Specified
        Time Spent - 0.5h
        17.
        Drop sqoop2 Sub-task Resolved Masatake Iwasaki

        100%

        Original Estimate - Not Specified Original Estimate - Not Specified
        Time Spent - 40m
        18.
        Drop Mahout Sub-task Resolved Masatake Iwasaki

        100%

        Original Estimate - Not Specified Original Estimate - Not Specified
        Time Spent - 40m
        19.
        Drop Giraph Sub-task Resolved Masatake Iwasaki

        100%

        Original Estimate - Not Specified Original Estimate - Not Specified
        Time Spent - 1h 10m
        20.
        Bump GPDB to 5.28.5 Sub-task Resolved Kengo Seki

        100%

        Original Estimate - Not Specified Original Estimate - Not Specified
        Time Spent - 0.5h
        21.
        Bump Livy to 0.7.1 Sub-task Resolved Kengo Seki

        100%

        Original Estimate - Not Specified Original Estimate - Not Specified
        Time Spent - 0.5h
        22.
        Drop Flume Sub-task Resolved Masatake Iwasaki

        100%

        Original Estimate - Not Specified Original Estimate - Not Specified
        Time Spent - 40m
        23.
        Drop ignite-hadoop Sub-task Resolved Masatake Iwasaki

        100%

        Original Estimate - Not Specified Original Estimate - Not Specified
        Time Spent - 1h 10m
        24.
        Drop QFS Sub-task Resolved Masatake Iwasaki

        100%

        Original Estimate - Not Specified Original Estimate - Not Specified
        Time Spent - 1.5h
        25.
        Drop Fedora 31 and add Fedora 33 support Sub-task Resolved Kengo Seki

        100%

        Original Estimate - Not Specified Original Estimate - Not Specified
        Time Spent - 1h 20m
        26.
        Drop Ubuntu 16.04 support Sub-task Resolved Kengo Seki

        100%

        Original Estimate - Not Specified Original Estimate - Not Specified
        Time Spent - 0.5h
        27.
        Bump Oozie to 5.2.1 Sub-task Resolved Masatake Iwasaki

        100%

        Original Estimate - Not Specified Original Estimate - Not Specified
        Time Spent - 40m
        28.
        Drop bigtop-tomcat Sub-task Resolved Masatake Iwasaki

        100%

        Original Estimate - Not Specified Original Estimate - Not Specified
        Time Spent - 40m
        29.
        Drop Debian 9 support Sub-task Resolved Kengo Seki

        100%

        Original Estimate - Not Specified Original Estimate - Not Specified
        Time Spent - 50m
        30.
        Bump Solr to v8.7.0 Sub-task Resolved Masatake Iwasaki

        100%

        Original Estimate - Not Specified Original Estimate - Not Specified
        Time Spent - 2.5h
        31.
        Bump ambari to v2.7.5 Sub-task Resolved Masatake Iwasaki

        100%

        Original Estimate - Not Specified Original Estimate - Not Specified
        Time Spent - 2h
        32.
        Minimum chnages for upgrading Bigtop-ambari-mpack aligned with Ambari 2.7.5 Sub-task Resolved Yuqi Gu

        100%

        Original Estimate - Not Specified Original Estimate - Not Specified
        Time Spent - 1h 50m

        Activity

          People

            yqGu Yuqi Gu
            sekikn Kengo Seki
            Votes:
            5 Vote for this issue
            Watchers:
            20 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Time Tracking

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0h
                0h
                Logged:
                Time Spent - 36h 40m
                36h 40m