Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-19848

Zookeeper thread leaks in hbase-spark bulkLoad method

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 1.2.0
    • 3.0.0-alpha-1
    • spark, Zookeeper
    • hbase-spark-1.2.0-cdh5.12.1 version

      spark 1.6

    • Reviewed
    • Patch, Important

    Description

      In hbase-spark project, HBaseContext provides bulkload methond for loading spark rdd data to hbase easily.But when i using it frequently, the program will throw "cannot create native thread" exception.

      using pstack command in spark driver process , the thread num is increasing 

      using jstack, named "main-SendThread" and "main-EventThread"  thread so many

      It seems like that , connection created before bulkload ,but close method uninvoked at last

      Attachments

        1. HBaseContext.scala
          41 kB
          Key Hutu
        2. HBaseContext.patch
          16 kB
          Key Hutu
        3. HBASE-19848-V2.patch
          1 kB
          Key Hutu
        4. HBASE-19848-V3.patch
          0.7 kB
          Key Hutu

        Activity

          huaxiang Hua Xiang added a comment -

          Hi Key Hutu, can you upload a patch? Thanks.

          huaxiang Hua Xiang added a comment - Hi  Key Hutu , can you upload a patch? Thanks.
          Key Hutu Key Hutu added a comment -

          At bulkload and hbaseBulkLoadThinRows method , do close()

          I submit a patch file in attachment

          Key Hutu Key Hutu added a comment - At bulkload and hbaseBulkLoadThinRows method , do close() I submit a patch file in attachment
          yuzhihong@gmail.com Ted Yu added a comment -

          You can use dev-support/submit-patch.py to submit patch to this JIRA.

          QA bot wouldn't accept the java file for test run.

          yuzhihong@gmail.com Ted Yu added a comment - You can use dev-support/submit-patch.py to submit patch to this JIRA. QA bot wouldn't accept the java file for test run.
          huaxiang Hua Xiang added a comment -

          Hi Key Hutu, as Ted mentioned, you can use the submit-patch script to submit the patch. Or use git diff to create a patch by 

          "git diff --no-prefix". yuzhihong@gmail.com and saint.ack@gmail.com, can you add Key Hutu to contributor list so he can assign the Jira to himself? Thanks.

          huaxiang Hua Xiang added a comment - Hi  Key Hutu , as Ted mentioned, you can use the submit-patch script to submit the patch. Or use git diff to create a patch by  "git diff --no-prefix". yuzhihong@gmail.com and saint.ack@gmail.com , can you add Key Hutu to contributor list so he can assign the Jira to himself? Thanks.
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 1m 54s Docker mode activated.
                Prechecks
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
                master Compile Tests
          +1 mvninstall 4m 16s master passed
          +1 compile 0m 56s master passed
          +1 scaladoc 0m 29s master passed
                Patch Compile Tests
          -1 compile 0m 26s hbase-spark in the patch failed.
          -1 scalac 0m 26s hbase-spark in the patch failed.
          -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
          -1 scaladoc 0m 29s hbase-spark generated 9 new + 0 unchanged - 0 fixed = 9 total (was 0)
                Other Tests
          -1 unit 0m 26s hbase-spark in the patch failed.
          +1 asflicense 0m 9s The patch does not generate ASF License warnings.
          9m 18s



          Subsystem Report/Notes
          Docker Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:eee3b01
          JIRA Issue HBASE-19848
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12908326/HBaseContext.patch
          Optional Tests asflicense scalac scaladoc unit compile
          uname Linux fabae75c577f 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018 x86_64 GNU/Linux
          Build tool maven
          Personality /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
          git revision master / 67f91f1282
          maven version: Apache Maven 3.5.2 (138edd61fd100ec658bfa2d307c43b76940a5d7d; 2017-10-18T07:58:13Z)
          compile https://builds.apache.org/job/PreCommit-HBASE-Build/11253/artifact/patchprocess/patch-compile-hbase-spark.txt
          scalac https://builds.apache.org/job/PreCommit-HBASE-Build/11253/artifact/patchprocess/patch-compile-hbase-spark.txt
          whitespace https://builds.apache.org/job/PreCommit-HBASE-Build/11253/artifact/patchprocess/whitespace-eol.txt
          scaladoc https://builds.apache.org/job/PreCommit-HBASE-Build/11253/artifact/patchprocess/diff-scaladoc-scaladoc-hbase-spark.txt
          unit https://builds.apache.org/job/PreCommit-HBASE-Build/11253/artifact/patchprocess/patch-unit-hbase-spark.txt
          Test Results https://builds.apache.org/job/PreCommit-HBASE-Build/11253/testReport/
          Max. process+thread count 83 (vs. ulimit of 1000)
          modules C: hbase-spark U: hbase-spark
          Console output https://builds.apache.org/job/PreCommit-HBASE-Build/11253/console
          Powered by Apache Yetus 0.7.0 http://yetus.apache.org

          This message was automatically generated.

          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 1m 54s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.       master Compile Tests +1 mvninstall 4m 16s master passed +1 compile 0m 56s master passed +1 scaladoc 0m 29s master passed       Patch Compile Tests -1 compile 0m 26s hbase-spark in the patch failed. -1 scalac 0m 26s hbase-spark in the patch failed. -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply -1 scaladoc 0m 29s hbase-spark generated 9 new + 0 unchanged - 0 fixed = 9 total (was 0)       Other Tests -1 unit 0m 26s hbase-spark in the patch failed. +1 asflicense 0m 9s The patch does not generate ASF License warnings. 9m 18s Subsystem Report/Notes Docker Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:eee3b01 JIRA Issue HBASE-19848 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12908326/HBaseContext.patch Optional Tests asflicense scalac scaladoc unit compile uname Linux fabae75c577f 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018 x86_64 GNU/Linux Build tool maven Personality /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh git revision master / 67f91f1282 maven version: Apache Maven 3.5.2 (138edd61fd100ec658bfa2d307c43b76940a5d7d; 2017-10-18T07:58:13Z) compile https://builds.apache.org/job/PreCommit-HBASE-Build/11253/artifact/patchprocess/patch-compile-hbase-spark.txt scalac https://builds.apache.org/job/PreCommit-HBASE-Build/11253/artifact/patchprocess/patch-compile-hbase-spark.txt whitespace https://builds.apache.org/job/PreCommit-HBASE-Build/11253/artifact/patchprocess/whitespace-eol.txt scaladoc https://builds.apache.org/job/PreCommit-HBASE-Build/11253/artifact/patchprocess/diff-scaladoc-scaladoc-hbase-spark.txt unit https://builds.apache.org/job/PreCommit-HBASE-Build/11253/artifact/patchprocess/patch-unit-hbase-spark.txt Test Results https://builds.apache.org/job/PreCommit-HBASE-Build/11253/testReport/ Max. process+thread count 83 (vs. ulimit of 1000) modules C: hbase-spark U: hbase-spark Console output https://builds.apache.org/job/PreCommit-HBASE-Build/11253/console Powered by Apache Yetus 0.7.0 http://yetus.apache.org This message was automatically generated.
          yuzhihong@gmail.com Ted Yu added a comment -

          There were several compilation errors, such as:

          [ERROR] /testptch/hbase/hbase-spark/src/main/scala/org/apache/hadoop/hbase/spark/HBaseContext.scala:875: error: type WriterBuilder is not a member of object org.apache.hadoop.hbase.regionserver.StoreFile
          [ERROR]       new StoreFile.WriterBuilder(conf, new CacheConfig(tempConf), new HFileSystem(fs))
          

          Please fix, compile locally before uploading the next patch.

          Include the JIRA number in the name of patch file.

          yuzhihong@gmail.com Ted Yu added a comment - There were several compilation errors, such as: [ERROR] /testptch/hbase/hbase-spark/src/main/scala/org/apache/hadoop/hbase/spark/HBaseContext.scala:875: error: type WriterBuilder is not a member of object org.apache.hadoop.hbase.regionserver.StoreFile [ERROR] new StoreFile.WriterBuilder(conf, new CacheConfig(tempConf), new HFileSystem(fs)) Please fix, compile locally before uploading the next patch. Include the JIRA number in the name of patch file.
          Key Hutu Key Hutu added a comment -

          Hi !Ted Yu, huaxiang

          I fix compile error locally.

          In the patch, just make sure the connection is closing, testing the patch in addition no need.  What do you think ? Thank you

          Key Hutu Key Hutu added a comment - Hi !Ted Yu,  huaxiang I fix compile error locally. In the patch, just make sure the connection is closing, testing the patch in addition no need.  What do you think ? Thank you
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 15s Docker mode activated.
                Prechecks
          -1 @author 0m 0s The patch appears to contain 1 @author tags which the community has agreed to not allow in code contributions.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
                master Compile Tests
          +1 mvninstall 5m 43s master passed
          +1 compile 0m 59s master passed
          +1 scaladoc 0m 29s master passed
                Patch Compile Tests
          +1 compile 0m 55s the patch passed
          +1 scalac 0m 55s the patch passed
          -1 whitespace 0m 0s The patch has 2 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
          +1 scaladoc 0m 28s the patch passed
                Other Tests
          +1 unit 4m 9s hbase-spark in the patch passed.
          +1 asflicense 0m 8s The patch does not generate ASF License warnings.
          13m 18s



          Subsystem Report/Notes
          Docker Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:eee3b01
          JIRA Issue HBASE-19848
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12908560/HBASE-19848-V2.patch
          Optional Tests asflicense scalac scaladoc unit compile
          uname Linux 415e94c94a1c 3.13.0-133-generic #182-Ubuntu SMP Tue Sep 19 15:49:21 UTC 2017 x86_64 GNU/Linux
          Build tool maven
          Personality /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
          git revision master / 2e6bc12441
          maven version: Apache Maven 3.5.2 (138edd61fd100ec658bfa2d307c43b76940a5d7d; 2017-10-18T07:58:13Z)
          @author https://builds.apache.org/job/PreCommit-HBASE-Build/11289/artifact/patchprocess/author-tags.txt
          whitespace https://builds.apache.org/job/PreCommit-HBASE-Build/11289/artifact/patchprocess/whitespace-eol.txt
          Test Results https://builds.apache.org/job/PreCommit-HBASE-Build/11289/testReport/
          modules C: hbase-spark U: hbase-spark
          Console output https://builds.apache.org/job/PreCommit-HBASE-Build/11289/console
          Powered by Apache Yetus 0.6.0 http://yetus.apache.org

          This message was automatically generated.

          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 15s Docker mode activated.       Prechecks -1 @author 0m 0s The patch appears to contain 1 @author tags which the community has agreed to not allow in code contributions. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.       master Compile Tests +1 mvninstall 5m 43s master passed +1 compile 0m 59s master passed +1 scaladoc 0m 29s master passed       Patch Compile Tests +1 compile 0m 55s the patch passed +1 scalac 0m 55s the patch passed -1 whitespace 0m 0s The patch has 2 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply +1 scaladoc 0m 28s the patch passed       Other Tests +1 unit 4m 9s hbase-spark in the patch passed. +1 asflicense 0m 8s The patch does not generate ASF License warnings. 13m 18s Subsystem Report/Notes Docker Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:eee3b01 JIRA Issue HBASE-19848 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12908560/HBASE-19848-V2.patch Optional Tests asflicense scalac scaladoc unit compile uname Linux 415e94c94a1c 3.13.0-133-generic #182-Ubuntu SMP Tue Sep 19 15:49:21 UTC 2017 x86_64 GNU/Linux Build tool maven Personality /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh git revision master / 2e6bc12441 maven version: Apache Maven 3.5.2 (138edd61fd100ec658bfa2d307c43b76940a5d7d; 2017-10-18T07:58:13Z) @author https://builds.apache.org/job/PreCommit-HBASE-Build/11289/artifact/patchprocess/author-tags.txt whitespace https://builds.apache.org/job/PreCommit-HBASE-Build/11289/artifact/patchprocess/whitespace-eol.txt Test Results https://builds.apache.org/job/PreCommit-HBASE-Build/11289/testReport/ modules C: hbase-spark U: hbase-spark Console output https://builds.apache.org/job/PreCommit-HBASE-Build/11289/console Powered by Apache Yetus 0.6.0 http://yetus.apache.org This message was automatically generated.
          yuzhihong@gmail.com Ted Yu added a comment -

          Can you remove the author line ?
          Please remove whitespace (tab).

          yuzhihong@gmail.com Ted Yu added a comment - Can you remove the author line ? Please remove whitespace (tab).
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 10s Docker mode activated.
                Prechecks
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
                master Compile Tests
          +1 mvninstall 3m 52s master passed
          +1 compile 0m 50s master passed
          +1 scaladoc 0m 27s master passed
                Patch Compile Tests
          +1 compile 0m 52s the patch passed
          +1 scalac 0m 52s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 scaladoc 0m 28s the patch passed
                Other Tests
          +1 unit 4m 21s hbase-spark in the patch passed.
          +1 asflicense 0m 9s The patch does not generate ASF License warnings.
          11m 27s



          Subsystem Report/Notes
          Docker Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:eee3b01
          JIRA Issue HBASE-19848
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12908972/HBASE-19848-V3.patch
          Optional Tests asflicense scalac scaladoc unit compile
          uname Linux c6ceb0be5c72 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 13:48:03 UTC 2016 x86_64 GNU/Linux
          Build tool maven
          Personality /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
          git revision master / 2f4d0b94bc
          maven version: Apache Maven 3.5.2 (138edd61fd100ec658bfa2d307c43b76940a5d7d; 2017-10-18T07:58:13Z)
          Test Results https://builds.apache.org/job/PreCommit-HBASE-Build/11361/testReport/
          Max. process+thread count 1036 (vs. ulimit of 10000)
          modules C: hbase-spark U: hbase-spark
          Console output https://builds.apache.org/job/PreCommit-HBASE-Build/11361/console
          Powered by Apache Yetus 0.7.0 http://yetus.apache.org

          This message was automatically generated.

          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 10s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.       master Compile Tests +1 mvninstall 3m 52s master passed +1 compile 0m 50s master passed +1 scaladoc 0m 27s master passed       Patch Compile Tests +1 compile 0m 52s the patch passed +1 scalac 0m 52s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 scaladoc 0m 28s the patch passed       Other Tests +1 unit 4m 21s hbase-spark in the patch passed. +1 asflicense 0m 9s The patch does not generate ASF License warnings. 11m 27s Subsystem Report/Notes Docker Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:eee3b01 JIRA Issue HBASE-19848 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12908972/HBASE-19848-V3.patch Optional Tests asflicense scalac scaladoc unit compile uname Linux c6ceb0be5c72 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 13:48:03 UTC 2016 x86_64 GNU/Linux Build tool maven Personality /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh git revision master / 2f4d0b94bc maven version: Apache Maven 3.5.2 (138edd61fd100ec658bfa2d307c43b76940a5d7d; 2017-10-18T07:58:13Z) Test Results https://builds.apache.org/job/PreCommit-HBASE-Build/11361/testReport/ Max. process+thread count 1036 (vs. ulimit of 10000) modules C: hbase-spark U: hbase-spark Console output https://builds.apache.org/job/PreCommit-HBASE-Build/11361/console Powered by Apache Yetus 0.7.0 http://yetus.apache.org This message was automatically generated.
          yuzhihong@gmail.com Ted Yu added a comment -

          Thanks for the patch, Key.

          yuzhihong@gmail.com Ted Yu added a comment - Thanks for the patch, Key.
          hudson Hudson added a comment -

          FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #4515 (See https://builds.apache.org/job/HBase-Trunk_matrix/4515/)
          HBASE-19848 Zookeeper thread leaks in hbase-spark bulkLoad method (Key (tedyu: rev 8143d5afa4a34c5f06a22e30b5017958b8c3f60c)

          • (edit) hbase-spark/src/main/scala/org/apache/hadoop/hbase/spark/HBaseContext.scala
          hudson Hudson added a comment - FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #4515 (See https://builds.apache.org/job/HBase-Trunk_matrix/4515/ ) HBASE-19848 Zookeeper thread leaks in hbase-spark bulkLoad method (Key (tedyu: rev 8143d5afa4a34c5f06a22e30b5017958b8c3f60c) (edit) hbase-spark/src/main/scala/org/apache/hadoop/hbase/spark/HBaseContext.scala
          Key Hutu Key Hutu added a comment -

          thanks for your help, Ted Yu, huaxiang sun, thanks

          Key Hutu Key Hutu added a comment - thanks for your help, Ted Yu, huaxiang sun, thanks

          People

            Key Hutu Key Hutu
            Key Hutu Key Hutu
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Time Tracking

                Estimated:
                Original Estimate - 72h
                72h
                Remaining:
                Remaining Estimate - 72h
                72h
                Logged:
                Time Spent - Not Specified
                Not Specified