Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
1.2.0
-
hbase-spark-1.2.0-cdh5.12.1 version
spark 1.6
-
Reviewed
-
Patch, Important
Description
In hbase-spark project, HBaseContext provides bulkload methond for loading spark rdd data to hbase easily.But when i using it frequently, the program will throw "cannot create native thread" exception.
using pstack command in spark driver process , the thread num is increasing
using jstack, named "main-SendThread" and "main-EventThread" thread so many
It seems like that , connection created before bulkload ,but close method uninvoked at last
Attachments
Attachments
- HBaseContext.scala
- 41 kB
- Key Hutu
- HBaseContext.patch
- 16 kB
- Key Hutu
- HBASE-19848-V2.patch
- 1 kB
- Key Hutu
- HBASE-19848-V3.patch
- 0.7 kB
- Key Hutu
Activity
At bulkload and hbaseBulkLoadThinRows method , do close()
I submit a patch file in attachment
You can use dev-support/submit-patch.py to submit patch to this JIRA.
QA bot wouldn't accept the java file for test run.
Hi Key Hutu, as Ted mentioned, you can use the submit-patch script to submit the patch. Or use git diff to create a patch by
"git diff --no-prefix". yuzhihong@gmail.com and saint.ack@gmail.com, can you add Key Hutu to contributor list so he can assign the Jira to himself? Thanks.
-1 overall |
Vote | Subsystem | Runtime | Comment |
---|---|---|---|
0 | reexec | 1m 54s | Docker mode activated. |
Prechecks | |||
+1 | @author | 0m 0s | The patch does not contain any @author tags. |
-1 | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. |
master Compile Tests | |||
+1 | mvninstall | 4m 16s | master passed |
+1 | compile | 0m 56s | master passed |
+1 | scaladoc | 0m 29s | master passed |
Patch Compile Tests | |||
-1 | compile | 0m 26s | hbase-spark in the patch failed. |
-1 | scalac | 0m 26s | hbase-spark in the patch failed. |
-1 | whitespace | 0m 0s | The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply |
-1 | scaladoc | 0m 29s | hbase-spark generated 9 new + 0 unchanged - 0 fixed = 9 total (was 0) |
Other Tests | |||
-1 | unit | 0m 26s | hbase-spark in the patch failed. |
+1 | asflicense | 0m 9s | The patch does not generate ASF License warnings. |
9m 18s |
This message was automatically generated.
There were several compilation errors, such as:
[ERROR] /testptch/hbase/hbase-spark/src/main/scala/org/apache/hadoop/hbase/spark/HBaseContext.scala:875: error: type WriterBuilder is not a member of object org.apache.hadoop.hbase.regionserver.StoreFile [ERROR] new StoreFile.WriterBuilder(conf, new CacheConfig(tempConf), new HFileSystem(fs))
Please fix, compile locally before uploading the next patch.
Include the JIRA number in the name of patch file.
Hi !Ted Yu, huaxiang
I fix compile error locally.
In the patch, just make sure the connection is closing, testing the patch in addition no need. What do you think ? Thank you
-1 overall |
Vote | Subsystem | Runtime | Comment |
---|---|---|---|
0 | reexec | 0m 15s | Docker mode activated. |
Prechecks | |||
-1 | @author | 0m 0s | The patch appears to contain 1 @author tags which the community has agreed to not allow in code contributions. |
-1 | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. |
master Compile Tests | |||
+1 | mvninstall | 5m 43s | master passed |
+1 | compile | 0m 59s | master passed |
+1 | scaladoc | 0m 29s | master passed |
Patch Compile Tests | |||
+1 | compile | 0m 55s | the patch passed |
+1 | scalac | 0m 55s | the patch passed |
-1 | whitespace | 0m 0s | The patch has 2 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply |
+1 | scaladoc | 0m 28s | the patch passed |
Other Tests | |||
+1 | unit | 4m 9s | hbase-spark in the patch passed. |
+1 | asflicense | 0m 8s | The patch does not generate ASF License warnings. |
13m 18s |
Subsystem | Report/Notes |
---|---|
Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:eee3b01 |
JIRA Issue | |
JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12908560/HBASE-19848-V2.patch |
Optional Tests | asflicense scalac scaladoc unit compile |
uname | Linux 415e94c94a1c 3.13.0-133-generic #182-Ubuntu SMP Tue Sep 19 15:49:21 UTC 2017 x86_64 GNU/Linux |
Build tool | maven |
Personality | /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh |
git revision | master / 2e6bc12441 |
maven | version: Apache Maven 3.5.2 (138edd61fd100ec658bfa2d307c43b76940a5d7d; 2017-10-18T07:58:13Z) |
@author | https://builds.apache.org/job/PreCommit-HBASE-Build/11289/artifact/patchprocess/author-tags.txt |
whitespace | https://builds.apache.org/job/PreCommit-HBASE-Build/11289/artifact/patchprocess/whitespace-eol.txt |
Test Results | https://builds.apache.org/job/PreCommit-HBASE-Build/11289/testReport/ |
modules | C: hbase-spark U: hbase-spark |
Console output | https://builds.apache.org/job/PreCommit-HBASE-Build/11289/console |
Powered by | Apache Yetus 0.6.0 http://yetus.apache.org |
This message was automatically generated.
-1 overall |
Vote | Subsystem | Runtime | Comment |
---|---|---|---|
0 | reexec | 0m 10s | Docker mode activated. |
Prechecks | |||
+1 | @author | 0m 0s | The patch does not contain any @author tags. |
-1 | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. |
master Compile Tests | |||
+1 | mvninstall | 3m 52s | master passed |
+1 | compile | 0m 50s | master passed |
+1 | scaladoc | 0m 27s | master passed |
Patch Compile Tests | |||
+1 | compile | 0m 52s | the patch passed |
+1 | scalac | 0m 52s | the patch passed |
+1 | whitespace | 0m 0s | The patch has no whitespace issues. |
+1 | scaladoc | 0m 28s | the patch passed |
Other Tests | |||
+1 | unit | 4m 21s | hbase-spark in the patch passed. |
+1 | asflicense | 0m 9s | The patch does not generate ASF License warnings. |
11m 27s |
Subsystem | Report/Notes |
---|---|
Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:eee3b01 |
JIRA Issue | |
JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12908972/HBASE-19848-V3.patch |
Optional Tests | asflicense scalac scaladoc unit compile |
uname | Linux c6ceb0be5c72 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 13:48:03 UTC 2016 x86_64 GNU/Linux |
Build tool | maven |
Personality | /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh |
git revision | master / 2f4d0b94bc |
maven | version: Apache Maven 3.5.2 (138edd61fd100ec658bfa2d307c43b76940a5d7d; 2017-10-18T07:58:13Z) |
Test Results | https://builds.apache.org/job/PreCommit-HBASE-Build/11361/testReport/ |
Max. process+thread count | 1036 (vs. ulimit of 10000) |
modules | C: hbase-spark U: hbase-spark |
Console output | https://builds.apache.org/job/PreCommit-HBASE-Build/11361/console |
Powered by | Apache Yetus 0.7.0 http://yetus.apache.org |
This message was automatically generated.
FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #4515 (See https://builds.apache.org/job/HBase-Trunk_matrix/4515/)
HBASE-19848 Zookeeper thread leaks in hbase-spark bulkLoad method (Key (tedyu: rev 8143d5afa4a34c5f06a22e30b5017958b8c3f60c)
- (edit) hbase-spark/src/main/scala/org/apache/hadoop/hbase/spark/HBaseContext.scala
Hi Key Hutu, can you upload a patch? Thanks.