Details
-
Bug
-
Status: Closed
-
Major
-
Resolution: Fixed
-
0.98.1, 1.0.0, 0.94.18
-
None
-
Reviewed
-
HideThrift servers should use framed/compact protocol to protect against buffer overflow (default disabled as they are breaking old clients)
- hbase.regionserver.thrift.framed = true
- hbase.regionserver.thrift.compact = trueShowThrift servers should use framed/compact protocol to protect against buffer overflow (default disabled as they are breaking old clients) - hbase.regionserver.thrift.framed = true - hbase.regionserver.thrift.compact = true
Description
Upstream thrift library has a know issue (THRIFT-601) causing the thrift server to crash with an Out-of-Memory Error when bogus requests are sent.
This reproduces when a very large request size is sent in the request header, making the thrift server to allocate a large memory segment leading to OOM.
LoadBalancer health checks are the first "candidate" for bogus requests
Thrift developers admit this is a known issue with TBinaryProtocol and their recommandation is to use TCompactProtocol/TFramedTransport but this requires all thrift clients to be updated (might not be feasible atm)
So we need a fix similar to CASSANDRA-475.
Attachments
Attachments
- HBASE-11052_trunk_v4.patch
- 4 kB
- Adrian Muraru
- HBASE-11052_trunk_v3.patch
- 4 kB
- Adrian Muraru
- HBASE-11052_trunk_v1.patch
- 4 kB
- Adrian Muraru
- HBASE-11052_0.94_v4.patch
- 6 kB
- Adrian Muraru
- HBASE-11052_0.94_v2.patch
- 5 kB
- Adrian Muraru
Issue Links
- is duplicated by
-
HBASE-11420 ThriftServer (version 1) may crash on OOME
- Closed
Activity
FAILURE: Integrated in hbase-0.96-hadoop2 #284 (See https://builds.apache.org/job/hbase-0.96-hadoop2/284/)
HBASE-11547 Backport HBASE-11052 to 0.96 'Sending random data crashes thrift service' (Adrian) (tedyu: rev 7d782b6acc9b50c912fb6ae81c5f50acf2faebc0)
- hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift2/ThriftServer.java
- hbase-common/src/main/resources/hbase-default.xml
- hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift/ThriftServerRunner.java
FAILURE: Integrated in hbase-0.96 #410 (See https://builds.apache.org/job/hbase-0.96/410/)
HBASE-11547 Backport HBASE-11052 to 0.96 'Sending random data crashes thrift service' (Adrian) (tedyu: rev 7d782b6acc9b50c912fb6ae81c5f50acf2faebc0)
- hbase-common/src/main/resources/hbase-default.xml
- hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift2/ThriftServer.java
- hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift/ThriftServerRunner.java
Oops, you're right.
This JIRA is in Closed state - let me open one for backporting.
FAILURE: Integrated in HBase-0.94-JDK7 #147 (See https://builds.apache.org/job/HBase-0.94-JDK7/147/)
HBASE-11052 Sending random data crashes thrift service (Adrian Muraru) (tedyu: rev ce722679a0a9096d253230ee0f61b9fff9c3638e)
- src/main/java/org/apache/hadoop/hbase/thrift/TBoundedThreadPoolServer.java
- src/main/java/org/apache/hadoop/hbase/thrift2/ThriftServer.java
- src/main/resources/hbase-default.xml
- src/main/java/org/apache/hadoop/hbase/thrift/ThriftServerRunner.java
FAILURE: Integrated in HBase-0.94 #1379 (See https://builds.apache.org/job/HBase-0.94/1379/)
HBASE-11052 Sending random data crashes thrift service (Adrian Muraru) (tedyu: rev ce722679a0a9096d253230ee0f61b9fff9c3638e)
- src/main/resources/hbase-default.xml
- src/main/java/org/apache/hadoop/hbase/thrift/TBoundedThreadPoolServer.java
- src/main/java/org/apache/hadoop/hbase/thrift/ThriftServerRunner.java
- src/main/java/org/apache/hadoop/hbase/thrift2/ThriftServer.java
SUCCESS: Integrated in HBase-0.94-security #493 (See https://builds.apache.org/job/HBase-0.94-security/493/)
HBASE-11052 Sending random data crashes thrift service (Adrian Muraru) (tedyu: rev ce722679a0a9096d253230ee0f61b9fff9c3638e)
- src/main/java/org/apache/hadoop/hbase/thrift/ThriftServerRunner.java
- src/main/java/org/apache/hadoop/hbase/thrift2/ThriftServer.java
- src/main/java/org/apache/hadoop/hbase/thrift/TBoundedThreadPoolServer.java
- src/main/resources/hbase-default.xml
FAILURE: Integrated in HBase-0.98-on-Hadoop-1.1 #325 (See https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/325/)
HBASE-11052 Sending random data crashes thrift service (Adrian Muraru) (tedyu: rev 1c4cdefe21b5bf45aac223a359e807845670bed3)
- hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift/ThriftServerRunner.java
- hbase-common/src/main/resources/hbase-default.xml
- hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift2/ThriftServer.java
FAILURE: Integrated in HBase-0.98 #344 (See https://builds.apache.org/job/HBase-0.98/344/)
HBASE-11052 Sending random data crashes thrift service (Adrian Muraru) (tedyu: rev 1c4cdefe21b5bf45aac223a359e807845670bed3)
- hbase-common/src/main/resources/hbase-default.xml
- hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift2/ThriftServer.java
- hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift/ThriftServerRunner.java
+1 for 0.98, hbase.regionserver.thrift.framed defaults to false.
SUCCESS: Integrated in HBase-TRUNK #5214 (See https://builds.apache.org/job/HBase-TRUNK/5214/)
HBASE-11052 Sending random data crashes thrift service (tedyu: rev 1324a3cb1807ee9eaa65deeefca5ee222045bdc3)
- hbase-common/src/main/resources/hbase-default.xml
- hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift/ThriftServerRunner.java
- hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift2/ThriftServer.java
Integrated to trunk.
Thanks for the patch, Adrian.
apurtell: do you want this in 0.98 ?
lhofhansl: do you want this in 0.94 ?
To be clear, we are not making the switch in the patch; framed/compact is false. On rolling upgrade old clients will still work? We just need to talk up need to switch? Thanks amuraru
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12648824/HBASE-11052_trunk_v4.patch
against trunk revision .
ATTACHMENT ID: 12648824
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 lineLengths. The patch does not introduce lines longer than 100
+1 site. The mvn site goal succeeds with this patch.
+1 core tests. The patch passed unit tests in .
Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/9717//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9717//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9717//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9717//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9717//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9717//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9717//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9717//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9717//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9717//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/9717//console
This message is automatically generated.
patch ver4 for trunk - reverting defaults in hbase-default.xml as they were making tests fail
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12648744/HBASE-11052_trunk_v3.patch
against trunk revision .
ATTACHMENT ID: 12648744
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 lineLengths. The patch does not introduce lines longer than 100
+1 site. The mvn site goal succeeds with this patch.
-1 core tests. The patch failed these unit tests:
org.apache.hadoop.hbase.thrift.TestThriftServerCmdLine
Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/9713//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9713//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9713//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9713//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9713//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9713//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9713//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9713//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9713//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9713//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/9713//console
This message is automatically generated.
Wrong patch format, resubmitted - jenkins is not happy with git format-patch
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12648735/HBASE-11052_0.94_v3.patch
against trunk revision .
ATTACHMENT ID: 12648735
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
-1 patch. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/9712//console
This message is automatically generated.
Looks good to me.
For 0.98 and 0.96 we could keep the default in the code for hbase.regionserver.thrift.compact and hbase.regionserver.thrift.framed as 'false' but add entries to hbase-default.xml for these as 'true' with a note in the description that changing it back to the old default exposes the service to DoS.
lgtm
+ transportFactory = new TFramedTransport.Factory(conf.getInt(MAX_FRAME_SIZE_CONF_KEY, 2) * 1024 * 1024);
Long line.
Trunk patch attached - both thrift v1 and v2 server are now using by default compact-protocol/framed-transport
v2 for 0.94 branch - HThrift2 is now using default compact-protocol/framed-transport
for 0.95 onwards we should consider configuring HThrift server to use compact/framed transport by default to avoid the OOM.
Any chance for a patch against trunk amuraru?
Note that thrift 0.9.0 removes (weird!) the message limit support in TBinaryProtocol (THRIFT-820), so for 0.95 onwards we should consider configuring HThrift server to use compact/framed transport by default to avoid the OOM.
THRIFT-2660should also help such cases generally.