Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-18175

Add hbase-spark integration test into hbase-spark-it

    XMLWordPrintableJSON

Details

    • Hide
      <!-- markdown -->
      HBase now ships with an integration test for our integration with Apache Spark.

      You can run this test on a cluster by using an equivalent to the below, e.g. if the version of HBase is 2.0.0-alpha-2

      ```
      spark-submit --class org.apache.hadoop.hbase.spark.IntegrationTestSparkBulkLoad HBASE_HOME/lib/hbase-spark-it-2.0.0-alpha-2-tests.jar -Dhbase.spark.bulkload.chainlength=500000 -m slowDeterministic
      ```
      Show
      <!-- markdown --> HBase now ships with an integration test for our integration with Apache Spark. You can run this test on a cluster by using an equivalent to the below, e.g. if the version of HBase is 2.0.0-alpha-2 ``` spark-submit --class org.apache.hadoop.hbase.spark.IntegrationTestSparkBulkLoad HBASE_HOME/lib/hbase-spark-it-2.0.0-alpha-2-tests.jar -Dhbase.spark.bulkload.chainlength=500000 -m slowDeterministic ```

    Description

      After HBASE-17574, all test under hbase-spark are regarded as unit test, and this jira will add integration test of hbase-spark into hbase-it. This patch run same tests as mapreduce.IntegrationTestBulkLoad, just change mapreduce to spark.

      test in Maven:
      mvn verify -Dit.test=IntegrationTestSparkBulkLoad

      test on cluster:
      spark-submit --class org.apache.hadoop.hbase.spark.IntegrationTestSparkBulkLoad HBASE_HOME/lib/hbase-it-2.0.0-SNAPSHOT-tests.jar -Dhbase.spark.bulkload.chainlength=500000 -m slowDeterministic

      Attachments

        1. addendum-master.patch
          1 kB
          Yi Liang
        2. hbase-18175-master-v2.patch
          37 kB
          Yi Liang
        3. hbase-18175-master-v3.patch
          37 kB
          Yi Liang
        4. hbase-18175-master-v4.patch
          37 kB
          Yi Liang
        5. hbase-18175-master-v5.patch
          38 kB
          Yi Liang
        6. hbase-18175-master-v6.patch
          38 kB
          Yi Liang
        7. hbase-18175-master-v7.patch
          38 kB
          Yi Liang
        8. hbase-18175-master-v8.patch
          38 kB
          Yi Liang
        9. hbase-18175-master-v9.patch
          38 kB
          Yi Liang
        10. hbase-18175-v1.patch
          25 kB
          Yi Liang

        Issue Links

          blocks

          New Feature - A new feature of the product, which has yet to be developed. HBASE-14160 backport hbase-spark module to branch-1 and branch-2

          • Major - Major loss of function.
          • Closed
          is depended upon by

          Task - A task that needs to be done. HBASE-14161 Add hbase-spark integration tests to IT jenkins job

          • Major - Major loss of function.
          • Closed

          Activity

            People

              easyliangjob Yi Liang
              easyliangjob Yi Liang
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: