Details
-
Task
-
Status: Closed
-
Major
-
Resolution: Fixed
-
2.0.0
-
None
-
Incompatible change
-
HideHTable class has been marked as private API before, and now it's no longer directly instantiable from client code (all public constructors have been removed). All clients should use Connection#getTable() and Connection#getRegionLocator() when appropriate to obtain Table and RegionLocator implementations to work with.ShowHTable class has been marked as private API before, and now it's no longer directly instantiable from client code (all public constructors have been removed). All clients should use Connection#getTable() and Connection#getRegionLocator() when appropriate to obtain Table and RegionLocator implementations to work with.
Description
Finish cleanup from HBASE-9117 removing old API. This issue covers tasks 6 and 7 from the list here: https://issues.apache.org/jira/browse/HBASE-9117?focusedCommentId=13919716&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13919716
To be done in master branch only.
Marked as beginner task. The idea is straight-forward. It is just a lot of work going through all tests converting to use new API.
Attachments
Attachments
- HBASE-12586.patch
- 56 kB
- Mikhail Antonov
- HBASE-12586.patch
- 56 kB
- Mikhail Antonov
- HBASE-12586.patch
- 55 kB
- Mikhail Antonov
- HBASE-12586.patch
- 54 kB
- Mikhail Antonov
- HBASE-12586.patch
- 49 kB
- Mikhail Antonov
Issue Links
- is related to
-
HBASE-9117 Remove HTablePool and all HConnection pooling related APIs
- Closed
-
HBASE-13197 Connection API cleanup
- Closed
Activity
I probably missed some discussions around..are there thoughts to remove HConnectionManager now?
Thanks Mikhail for picking this up. See HBASE-10602 and all related issues for an overview of the changes. In summary, ConnectionFactory replaces HConnectionManager.
Just to catch up on that..
- As I see, HTableInterface is audience-private, deprecated class - could/should it be removed in 2.0?
- What's interesting, HTableInterfaceFactory is audience-public, although it's used only to deal with HTableInterface's (audience-private). And though it's public/stable, it isn't not used anywhere in the code but in HTablePool, as far as I can tell. I'm not sure we could easily remove this class?
Remove both for 2.0 mantonov. That HTIF is public is oversight i'd say.
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12703199/HBASE-12586.patch
against master branch at commit 95fc53f1056238e0b5f65df7f3015f2aec3d0a20.
ATTACHMENT ID: 12703199
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 27 new or modified tests.
-1 javac. The patch appears to cause mvn compile goal to fail with Hadoop version 2.4.1.
Compilation errors resume:
[ERROR] COMPILATION ERROR :
[ERROR] /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/hbase-client/src/main/java/org/apache/hadoop/hbase/client/HTableFactory.java:[41,14] no suitable constructor found for HTable(org.apache.hadoop.conf.Configuration,org.apache.hadoop.hbase.TableName)
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.2:compile (default-compile) on project hbase-client: Compilation failure
[ERROR] /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/hbase-client/src/main/java/org/apache/hadoop/hbase/client/HTableFactory.java:[41,14] no suitable constructor found for HTable(org.apache.hadoop.conf.Configuration,org.apache.hadoop.hbase.TableName)
[ERROR] constructor org.apache.hadoop.hbase.client.HTable.HTable(org.apache.hadoop.hbase.client.ClusterConnection,org.apache.hadoop.hbase.client.BufferedMutatorParams) is not applicable
[ERROR] (actual argument org.apache.hadoop.conf.Configuration cannot be converted to org.apache.hadoop.hbase.client.ClusterConnection by method invocation conversion)
[ERROR] constructor org.apache.hadoop.hbase.client.HTable.HTable(org.apache.hadoop.hbase.TableName,org.apache.hadoop.hbase.client.ClusterConnection,org.apache.hadoop.hbase.client.TableConfiguration,org.apache.hadoop.hbase.client.RpcRetryingCallerFactory,org.apache.hadoop.hbase.ipc.RpcControllerFactory,java.util.concurrent.ExecutorService) is not applicable
[ERROR] (actual and formal argument lists differ in length)
[ERROR] -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn <goals> -rf :hbase-client
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/13126//console
This message is automatically generated.
Incorrectly-exported patch, removed files didn't get included.. Updating.
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12703205/HBASE-12586.patch
against master branch at commit 95fc53f1056238e0b5f65df7f3015f2aec3d0a20.
ATTACHMENT ID: 12703205
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 27 new or modified tests.
+1 hadoop versions. The patch compiles with all supported hadoop versions (2.4.1 2.5.2 2.6.0)
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
-1 javadoc. The javadoc tool appears to have generated 1 warning messages.
-1 checkstyle. The applied patch generated 1931 checkstyle errors (more than the master's current 1930 errors).
+1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 lineLengths. The patch does not introduce lines longer than 100
-1 site. The patch appears to cause mvn site goal to fail.
-1 core tests. The patch failed these unit tests:
org.apache.hadoop.hbase.regionserver.wal.TestLogRollPeriod
org.apache.hadoop.hbase.master.handler.TestTableDescriptorModification
org.apache.hadoop.hbase.master.cleaner.TestSnapshotFromMaster
org.apache.hadoop.hbase.master.TestMasterRestartAfterDisablingTable
org.apache.hadoop.hbase.regionserver.wal.TestWALReplay
org.apache.hadoop.hbase.regionserver.TestJoinedScanners
org.apache.hadoop.hbase.security.visibility.TestVisibilityLabelsWithACL
org.apache.hadoop.hbase.mapred.TestTableMapReduceUtil
org.apache.hadoop.hbase.master.TestGetLastFlushedSequenceId
org.apache.hadoop.hbase.master.TestMaster
org.apache.hadoop.hbase.regionserver.TestTags
org.apache.hadoop.hbase.security.access.TestCellACLWithMultipleVersions
org.apache.hadoop.hbase.security.access.TestAccessController
org.apache.hadoop.hbase.quotas.TestQuotaAdmin
org.apache.hadoop.hbase.regionserver.TestRegionMergeTransactionOnCluster
org.apache.hadoop.hbase.mapreduce.TestImportTsv
org.apache.hadoop.hbase.client.TestTimestampsFilter
org.apache.hadoop.hbase.TestMetaTableAccessorNoCluster
org.apache.hadoop.hbase.replication.TestReplicationKillMasterRS
org.apache.hadoop.hbase.replication.regionserver.TestRegionReplicaReplicationEndpoint
org.apache.hadoop.hbase.mapred.TestTableSnapshotInputFormat
org.apache.hadoop.hbase.security.access.TestAccessController2
org.apache.hadoop.hbase.trace.TestHTraceHooks
org.apache.hadoop.hbase.master.TestRegionPlacement
org.apache.hadoop.hbase.regionserver.TestCompactionState
org.apache.hadoop.hbase.mapreduce.TestImportTSVWithTTLs
org.apache.hadoop.hbase.fs.TestBlockReorder
org.apache.hadoop.hbase.regionserver.wal.TestLogRolling
org.apache.hadoop.hbase.util.TestMiniClusterLoadParallel
org.apache.hadoop.hbase.filter.TestMultiRowRangeFilter
org.apache.hadoop.hbase.regionserver.TestEndToEndSplitTransaction
org.apache.hadoop.hbase.util.TestHBaseFsck
org.apache.hadoop.hbase.replication.TestReplicationDisableInactivePeer
org.apache.hadoop.hbase.mapreduce.TestTableInputFormatScan2
org.apache.hadoop.hbase.util.hbck.TestOfflineMetaRebuildHole
org.apache.hadoop.hbase.replication.regionserver.TestReplicationSink
org.apache.hadoop.hbase.mapreduce.TestTableMapReduce
org.apache.hadoop.hbase.replication.TestReplicationKillSlaveRS
org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster
org.apache.hadoop.hbase.mapreduce.TestTimeRangeMapRed
org.apache.hadoop.hbase.filter.TestColumnRangeFilter
org.apache.hadoop.hbase.mapreduce.TestCopyTable
org.apache.hadoop.hbase.util.TestMergeTable
org.apache.hadoop.hbase.master.TestMasterFailover
org.apache.hadoop.hbase.mapreduce.TestMultithreadedTableMapper
org.apache.hadoop.hbase.replication.regionserver.TestRegionReplicaReplicationEndpointNoMaster
org.apache.hadoop.hbase.io.encoding.TestLoadAndSwitchEncodeOnDisk
org.apache.hadoop.hbase.filter.TestFuzzyRowAndColumnRangeFilter
org.apache.hadoop.hbase.security.visibility.TestVisibilityLabelsReplication
org.apache.hadoop.hbase.namespace.TestNamespaceAuditor
org.apache.hadoop.hbase.mapreduce.TestTableInputFormatScan1
org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFilesSplitRecovery
org.apache.hadoop.hbase.regionserver.TestRegionReplicaFailover
org.apache.hadoop.hbase.mapred.TestTableInputFormat
org.apache.hadoop.hbase.util.TestRegionSplitter
org.apache.hadoop.hbase.regionserver.TestHRegionServerBulkLoad
org.apache.hadoop.hbase.security.visibility.TestVisibilityWithCheckAuths
org.apache.hadoop.hbase.regionserver.TestScannerWithBulkload
org.apache.hadoop.hbase.filter.TestFilterWithScanLimits
org.apache.hadoop.hbase.security.access.TestAccessControlFilter
org.apache.hadoop.hbase.backup.TestHFileArchiving
org.apache.hadoop.hbase.security.visibility.TestEnforcingScanLabelGenerator
org.apache.hadoop.hbase.master.TestMasterOperationsForRegionReplicas
org.apache.hadoop.hbase.regionserver.TestServerCustomProtocol
org.apache.hadoop.hbase.regionserver.TestRegionReplicas
org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFilesUseSecurityEndPoint
org.apache.hadoop.hbase.mapreduce.TestSecureLoadIncrementalHFiles
org.apache.hadoop.hbase.regionserver.TestRegionFavoredNodes
org.apache.hadoop.hbase.mapreduce.TestTableInputFormat
org.apache.hadoop.hbase.quotas.TestQuotaTableUtil
org.apache.hadoop.hbase.mapreduce.TestImportExport
org.apache.hadoop.hbase.filter.TestScanRowPrefix
org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat2
org.apache.hadoop.hbase.util.TestHBaseFsckEncryption
org.apache.hadoop.hbase.security.visibility.TestVisibilityLabelReplicationWithExpAsString
org.apache.hadoop.hbase.master.TestMasterFailoverBalancerPersistence
org.apache.hadoop.hbase.master.handler.TestTableDeleteFamilyHandler
org.apache.hadoop.hbase.regionserver.wal.TestWALReplayCompressed
org.apache.hadoop.hbase.security.visibility.TestDefaultScanLabelGeneratorStack
org.apache.hadoop.hbase.security.access.TestNamespaceCommands
org.apache.hadoop.hbase.mapreduce.TestCellCounter
org.apache.hadoop.hbase.master.TestRollingRestart
org.apache.hadoop.hbase.regionserver.compactions.TestCompactionWithThroughputController
org.apache.hadoop.hbase.mapreduce.TestWALPlayer
org.apache.hadoop.hbase.mapreduce.TestHRegionPartitioner
org.apache.hadoop.hbase.regionserver.wal.TestLogRollAbort
org.apache.hadoop.hbase.filter.TestFilterWrapper
org.apache.hadoop.hbase.master.TestTableStateManager
org.apache.hadoop.hbase.TestAcidGuarantees
org.apache.hadoop.hbase.TestHBaseTestingUtility
org.apache.hadoop.hbase.replication.TestReplicationSyncUpTool
org.apache.hadoop.hbase.util.hbck.TestOfflineMetaRebuildBase
org.apache.hadoop.hbase.mapreduce.TestImportTSVWithVisibilityLabels
org.apache.hadoop.hbase.TestFullLogReconstruction
org.apache.hadoop.hbase.io.encoding.TestChangingEncoding
org.apache.hadoop.hbase.TestInfoServers
org.apache.hadoop.hbase.security.visibility.TestVisibilityLabelsWithDeletes
org.apache.hadoop.hbase.replication.TestPerTableCFReplication
org.apache.hadoop.hbase.security.visibility.TestVisibilityLabelsWithSLGStack
org.apache.hadoop.hbase.regionserver.TestRegionServerOnlineConfigChange
org.apache.hadoop.hbase.mapred.TestTableMapReduce
org.apache.hadoop.hbase.master.TestDistributedLogSplitting
org.apache.hadoop.hbase.replication.TestReplicationWithTags
org.apache.hadoop.hbase.util.TestMiniClusterLoadEncoded
org.apache.hadoop.hbase.regionserver.TestHRegion
org.apache.hadoop.hbase.regionserver.TestRegionServerNoMaster
org.apache.hadoop.hbase.regionserver.TestHRegionOnCluster
org.apache.hadoop.hbase.security.visibility.TestVisibilityLabelsWithCustomVisLabService
org.apache.hadoop.hbase.replication.TestMultiSlaveReplication
org.apache.hadoop.hbase.mapreduce.TestRowCounter
org.apache.hadoop.hbase.master.TestRestartCluster
org.apache.hadoop.hbase.util.TestCoprocessorScanPolicy
org.apache.hadoop.hbase.mapreduce.TestImportTSVWithOperationAttributes
org.apache.hadoop.hbase.replication.TestReplicationEndpoint
org.apache.hadoop.hbase.master.TestTableLockManager
org.apache.hadoop.hbase.client.TestCloneSnapshotFromClient
org.apache.hadoop.hbase.security.access.TestScanEarlyTermination
org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFiles
org.apache.hadoop.hbase.mapreduce.TestMultiTableInputFormat
org.apache.hadoop.hbase.master.TestMasterTransitions
org.apache.hadoop.hbase.replication.TestReplicationKillMasterRSCompressed
org.apache.hadoop.hbase.replication.TestReplicationSmallTests
org.apache.hadoop.hbase.replication.TestReplicationChangingPeerRegionservers
org.apache.hadoop.hbase.mapreduce.TestTableSnapshotInputFormat
org.apache.hadoop.hbase.mapreduce.TestSecureLoadIncrementalHFilesSplitRecovery
org.apache.hadoop.hbase.TestRegionRebalancing
org.apache.hadoop.hbase.regionserver.TestRegionServerMetrics
org.apache.hadoop.hbase.security.visibility.TestVisibilityLabelsWithDistributedLogReplay
org.apache.hadoop.hbase.security.visibility.TestVisibilityLabelsWithDefaultVisLabelService
org.apache.hadoop.hbase.security.visibility.TestVisibilityLablesWithGroups
org.apache.hadoop.hbase.TestMetaTableAccessor
org.apache.hadoop.hbase.master.TestAssignmentListener
org.apache.hadoop.hbase.quotas.TestQuotaThrottle
org.apache.hadoop.hbase.regionserver.wal.TestSecureWALReplay
org.apache.hadoop.hbase.master.handler.TestCreateTableHandler
org.apache.hadoop.hbase.regionserver.TestSCVFWithMiniCluster
org.apache.hadoop.hbase.regionserver.TestFSErrorsExposed
org.apache.hadoop.hbase.security.access.TestTablePermissions
org.apache.hadoop.hbase.regionserver.TestEncryptionRandomKeying
org.apache.hadoop.hbase.util.TestMiniClusterLoadSequential
org.apache.hadoop.hbase.replication.TestMasterReplication
org.apache.hadoop.hbase.regionserver.TestEncryptionKeyRotation
org.apache.hadoop.hbase.master.handler.TestEnableTableHandler
org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat
org.apache.hadoop.hbase.wal.TestWALFiltering
org.apache.hadoop.hbase.regionserver.TestPerColumnFamilyFlush
org.apache.hadoop.hbase.util.hbck.TestOfflineMetaRebuildOverlap
-1 core zombie tests. There are 4 zombie test(s): at org.apache.hadoop.hbase.ResourceCheckerJUnitListener.testStarted(ResourceCheckerJUnitListener.java:178)
at org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster.testSSHWaitForServerToAssignRegion(TestAssignmentManagerOnCluster.java:855)
at org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster.testAssignRacingWithSSH(TestAssignmentManagerOnCluster.java:839)
at org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster.testCloseHang(TestAssignmentManagerOnCluster.java:693)
at org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster.testMoveRegionOfDeletedTable(TestAssignmentManagerOnCluster.java:400)
at org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster.testAssignDisabledRegionBySSH(TestAssignmentManagerOnCluster.java:1088)
at org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster.testOpenFailed(TestAssignmentManagerOnCluster.java:561)
at org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster.testOpenFailedUnrecoverable(TestAssignmentManagerOnCluster.java:610)
at org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster.testCloseFailed(TestAssignmentManagerOnCluster.java:520)
at org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster.testAssignRegion(TestAssignmentManagerOnCluster.java:206)
at org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster.testAssignRegionOnRestartedServer(TestAssignmentManagerOnCluster.java:263)
at org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster.testMoveRegion(TestAssignmentManagerOnCluster.java:354)
at org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster.testReportRegionStateTransition(TestAssignmentManagerOnCluster.java:1123)
at org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster.testAssignOfflinedRegionBySSH(TestAssignmentManagerOnCluster.java:1011)
at org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster.testAssignWhileClosing(TestAssignmentManagerOnCluster.java:474)
at org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster.testOfflineRegion(TestAssignmentManagerOnCluster.java:304)
at org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster.testSSHWaitForServerToAssignRegion(TestAssignmentManagerOnCluster.java:889)
at org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster.testAssignRacingWithSSH(TestAssignmentManagerOnCluster.java:839)
at org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster.testCloseHang(TestAssignmentManagerOnCluster.java:693)
at org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster.testMoveRegionOfDeletedTable(TestAssignmentManagerOnCluster.java:400)
at org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster.testAssignDisabledRegionBySSH(TestAssignmentManagerOnCluster.java:1088)
at org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster.testOpenFailed(TestAssignmentManagerOnCluster.java:561)
at org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster.testOpenFailedUnrecoverable(TestAssignmentManagerOnCluster.java:610)
at org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster.testCloseFailed(TestAssignmentManagerOnCluster.java:520)
at org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster.testAssignRegion(TestAssignmentManagerOnCluster.java:206)
at org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster.testAssignRegionOnRestartedServer(TestAssignmentManagerOnCluster.java:263)
Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/13128//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13128//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13128//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13128//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13128//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13128//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13128//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13128//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13128//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13128//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13128//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13128//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Checkstyle Errors: https://builds.apache.org/job/PreCommit-HBASE-Build/13128//artifact/patchprocess/checkstyle-aggregate.html
Javadoc warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13128//artifact/patchprocess/patchJavadocWarnings.txt
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/13128//console
This message is automatically generated.
+1 on patches that remove more code than they add. This one could be a little contrary to get in mantonov... lots of presumption out in test land.
stack Yeah... contrary in sense - "a lot of tests needs to be updated", or at more fundamental level? I hope to get tests fixed pretty soon and update the patch. This patch doesn't yet include changes regarding
{delete, get}Connection methods - that's next.
The former... "a lot of tests needs to be updated" This is good work.
Turns out the vast majority of broken tests (all but 3, which are easy to fix separately) are fixable with changing ConnectionManager#getConnectionInternal() to use unmanaged connections. That somewhat diminishes the role of internal connection cache, but I believe connection caching going away anyway (as stated in javadoc)?
Thoughts?
mantonov Sounds good. Suggest check how long hadoopqa takes to run. Compare it to your patch that swaps in a bunch of unmanaged uses. Is it much longer? If not radically different, can address bad cases in a followup I'd say.
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12703480/HBASE-12586.patch
against master branch at commit 948746ce4ed3bd174927c41bd4884cad70d693ef.
ATTACHMENT ID: 12703480
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 33 new or modified tests.
+1 hadoop versions. The patch compiles with all supported hadoop versions (2.4.1 2.5.2 2.6.0)
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
-1 javadoc. The javadoc tool appears to have generated 1 warning messages.
-1 checkstyle. The applied patch generated 1928 checkstyle errors (more than the master's current 1927 errors).
+1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 lineLengths. The patch does not introduce lines longer than 100
-1 site. The patch appears to cause mvn site goal to fail.
+1 core tests. The patch passed unit tests in .
Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/13144//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13144//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13144//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13144//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13144//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13144//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13144//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13144//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13144//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13144//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13144//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13144//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Checkstyle Errors: https://builds.apache.org/job/PreCommit-HBASE-Build/13144//artifact/patchprocess/checkstyle-aggregate.html
Javadoc warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13144//artifact/patchprocess/patchJavadocWarnings.txt
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/13144//console
This message is automatically generated.
The build took 2 hr 15 min, no measurable difference from previous runs on that executor.
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12703535/HBASE-12586.patch
against master branch at commit 5025d3aa91d18310fc4d738114ee2b58e48c46c2.
ATTACHMENT ID: 12703535
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 33 new or modified tests.
+1 hadoop versions. The patch compiles with all supported hadoop versions (2.4.1 2.5.2 2.6.0)
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 checkstyle. The applied patch does not increase the total number of checkstyle errors
+1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 lineLengths. The patch does not introduce lines longer than 100
-1 site. The patch appears to cause mvn site goal to fail.
+1 core tests. The patch passed unit tests in .
Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/13147//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13147//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13147//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13147//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13147//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13147//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13147//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13147//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13147//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13147//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13147//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13147//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html
Checkstyle Errors: https://builds.apache.org/job/PreCommit-HBASE-Build/13147//artifact/patchprocess/checkstyle-aggregate.html
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/13147//console
This message is automatically generated.
Site generation failed with OutOfMemoryError: PermGen space, not related.
Site generation failed with OutOfMemoryError: PermGen space, not related.
Thanks for looking into this. I've added -XX:MaxPermSize=256m to the maven jenkins env. Let's see whether this fixes the problem.
Site failure is very likely unrelated. Misty just fixed it. No harm in letting next hadoopqa run complete before commit. Good on you mantonov
Thinking more about it, I'd probably split this remaining work on API cleanup into several pieces, which I'd like to discuss here (or is it better to move it to other place? please excuse me if I'm bringing up points discussed elsewhere..).
One is this (removing public HTable constructors).
The second one is the general notion of cached/managed connection. Javadoc in ConnectionManager suggests that connection caching is going away. So shall we remove LRU cache of them from CM for 2.0? And thinking more long-term..are we going to get rid of notion of managed connections?
The other one is about number of methods to acquire connections. Currently we have ConnectionManager, which has a bunch of deprecated methods (to be replaced with ConnectionFactory calls) and also encapsulated HConnectionImplementation (which is directly accessed and constructed from ConnectionFactory), and we also have HConnectionManager class (the naming would suggest it's an impl of ConnectionManager, but it's totally unrelated), which is also deprecated. Would the following make sense as an aggressive, yet meaningful cleanup? or may be parts of that could be accommodated:
- Take HConnectionImplementation out as a top-level class
- Remove HConnectionManager
- Migrate code using ConnectionManager to use ConnectionFactory when possible
Discussion here is fine mantonov
..are we going to get rid of notion of managed connections?
Yes. Consensus was it was a bad idea. That said, their setup costs so especially in stuff like tests, reuse if possible.
On your aggressive cleanup, that sounds right to me. I am unclear on what #3 would mean so +1 on #1 and #2 but need more on #3.
Good on you mantonov Make subtasks off this issue and piecemeal it if that is easier to do?
stack this jira is about specific API cleanup, so I'd probably leave this one as is, and create one more jira (like we had for removing connection pooling), and then as I go thru #1 and #2 above I'll slice more subtasks, how does it sound?
#3 was meant that, as I see, we way want to reduce number of ways to acquire connections, ultimately to one So maybe merge ConnectionManager inside ConnectionFactory and get rid of the former (not sure yet, just a thought).
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12703851/HBASE-12586.patch
against master branch at commit b436db7d70c8a90b0167dc0e0120f503efb37e3c.
ATTACHMENT ID: 12703851
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 33 new or modified tests.
+1 hadoop versions. The patch compiles with all supported hadoop versions (2.4.1 2.5.2 2.6.0)
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 checkstyle. The applied patch does not increase the total number of checkstyle errors
+1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 lineLengths. The patch does not introduce lines longer than 100
-1 site. The patch appears to cause mvn site goal to fail.
+1 core tests. The patch passed unit tests in .
Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/13174//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13174//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13174//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13174//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13174//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13174//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13174//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13174//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13174//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13174//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13174//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13174//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Checkstyle Errors: https://builds.apache.org/job/PreCommit-HBASE-Build/13174//artifact/patchprocess/checkstyle-aggregate.html
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/13174//console
This message is automatically generated.
I'm seeing the site target completed successfully, but then permgen error pops up again.
+1 from me. Marked as incompatible change. You good w/ this ndimiduk?
A release note when you get a chance mantonov please.
Thanks stack, I'll put a release note. The next steps on connection cleanup are tracked in HBASE-13197 and subtasks (1 subtask so far).
mantonov great cleanup.
HTable has getRegionLocations, may be we need new jira for cleaning up those methods too?
(they marked deprecated and seems most tests and code that use HTable actually does that
only for this method).
thanks.
octo47 that applies to #isTableEnabled() and a host of other methods in HTable, too. Shall we add it as a subtask to HBASE-13197? Not technically part of connection API, but closely related IMO.
agree, thats not a public api, so subtask of HBASE-13197 will suite well.
This is great work mantonov.
Rename HTableWrapper to TableWrapper?
Can the concept of managed connections be removed after this patch?
I'm surprised the tests aren't taking longer now that they're setting up connections each time. Maybe there's some magic still going on down in the bowels?
ndimiduk are you talking about some particular tests?
actually many tests use connection from HBaseTestingUtility which is effectively a singleton.
ndimiduk yeah, I'd probably make renames in subsequent patches under HBASE-13197. After this patch we still have notion of managed connectins, ConnectionManager#CONNECTION_INSTANCES is still there, but we got closer to be able to remove it. I think in subtask of that jira mentioned above we can do that.
Regarding performance - besides what octo47 said, in tests we're putting up and shutting down miniclusters, which is far more expensive operation than establishing connections, also creating connection in minicluster environment (where we have in-memory ZK minicluster and all other things in process) is cheaper than in distributed cluster.. I think that's the reason.
Thanks stack! Let me rebase patches for HBASE-13198 and HBASE-13248 then.
FAILURE: Integrated in HBase-TRUNK #6265 (See https://builds.apache.org/job/HBase-TRUNK/6265/)
HBASE-12586 Task 6 & 7 from HBASE-9117, delete all public HTable constructors and delete ConnectionManager#
Connection (Mikhail Antonov) (stack: rev c5aca1919d774d2145005b748b3430a4b569922c)
- hbase-server/src/main/java/org/apache/hadoop/hbase/mapred/TableInputFormatBase.java
- hbase-client/src/main/java/org/apache/hadoop/hbase/client/HTableFactory.java
- hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestMasterReplication.java
- hbase-client/src/main/java/org/apache/hadoop/hbase/client/HTableInterfaceFactory.java
- hbase-client/src/main/java/org/apache/hadoop/hbase/CoprocessorEnvironment.java
- hbase-client/src/main/java/org/apache/hadoop/hbase/client/HTable.java
- hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestHTableWrapper.java
- hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestHCM.java
- hbase-server/src/main/java/org/apache/hadoop/hbase/client/HTableWrapper.java
- hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestTableInputFormat.java
- hbase-server/src/test/java/org/apache/hadoop/hbase/security/visibility/ExpAsStringVisibilityLabelServiceImpl.java
- hbase-server/src/test/java/org/apache/hadoop/hbase/client/HConnectionTestingUtility.java
- hbase-server/src/test/java/org/apache/hadoop/hbase/mapred/TestTableInputFormat.java
- hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsck.java
- hbase-client/src/main/java/org/apache/hadoop/hbase/client/ConnectionManager.java
- hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestSplitTransactionOnCluster.java
- hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
- hbase-client/src/main/java/org/apache/hadoop/hbase/MetaTableAccessor.java
- hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorHost.java
- hbase-server/src/main/java/org/apache/hadoop/hbase/util/ConnectionCache.java
Picking this one up.