[HBASE-11492] Hadoop configuration overrides some ipc parameters including tcpNoDelay - ASF JIRA

Details

Type: Bug
Status: Closed
Priority: Critical
Resolution: Fixed
Affects Version/s: 0.98.0, 0.99.0
Fix Version/s: 0.99.0, 0.98.4
Component/s: regionserver
Labels:
None

Hadoop Flags:

Reviewed
Release Note:

Hide
If the Hadoop configuration is read after the HBase configuration, Hadoop's settings can override HBase's settings if the names of the settings are the same. To avoid the risk of override, HBase has renamed the following settings (by prepending 'hbase.') so that you can set them independent of your setting for Hadoop. If you do not use the HBase-specific variants, the Hadoop settings will be used.

Old Name --> New Name
======================================================
ipc.server.listen.queue.size --> hbase.ipc.server.listen.queue.size
ipc.server.max.callqueue.size --> hbase.ipc.server.max.callqueue.size
ipc.server.callqueue.handler.factor --> hbase.ipc.server.callqueue.handler.factor
ipc.server.callqueue.read.share --> hbase.ipc.server.callqueue.read.share
ipc.server.callqueue.type --> hbase.ipc.server.callqueue.type
ipc.server.queue.max.call.delay --> hbase.ipc.server.queue.max.call.delay
ipc.server.max.callqueue.length --> hbase.ipc.server.max.callqueue.length
ipc.server.read.threadpool.size --> hbase.ipc.server.read.threadpool.size
ipc.server.tcpkeepalive --> hbase.ipc.server.tcpkeepalive
ipc.server.tcpnodelay --> hbase.ipc.server.tcpnodelay
ipc.client.call.purge.timeout --> hbase.ipc.client.call.purge.timeout
ipc.client.connection.maxidletime --> hbase.ipc.client.connection.maxidletime
ipc.client.idlethreshold --> hbase.ipc.client.idlethreshold
ipc.client.kill.max --> hbase.ipc.client.kill.max
ipc.server.scan.vtime.weight --> hbase.ipc.server.scan.vtime.weight

Show
If the Hadoop configuration is read after the HBase configuration, Hadoop's settings can override HBase's settings if the names of the settings are the same. To avoid the risk of override, HBase has renamed the following settings (by prepending 'hbase.') so that you can set them independent of your setting for Hadoop. If you do not use the HBase-specific variants, the Hadoop settings will be used. Old Name --> New Name ====================================================== ipc.server.listen.queue.size --> hbase.ipc.server.listen.queue.size ipc.server.max.callqueue.size --> hbase.ipc.server.max.callqueue.size ipc.server.callqueue.handler.factor --> hbase.ipc.server.callqueue.handler.factor ipc.server.callqueue.read.share --> hbase.ipc.server.callqueue.read.share ipc.server.callqueue.type --> hbase.ipc.server.callqueue.type ipc.server.queue.max.call.delay --> hbase.ipc.server.queue.max.call.delay ipc.server.max.callqueue.length --> hbase.ipc.server.max.callqueue.length ipc.server.read.threadpool.size --> hbase.ipc.server.read.threadpool.size ipc.server.tcpkeepalive --> hbase.ipc.server.tcpkeepalive ipc.server.tcpnodelay --> hbase.ipc.server.tcpnodelay ipc.client.call.purge.timeout --> hbase.ipc.client.call.purge.timeout ipc.client.connection.maxidletime --> hbase.ipc.client.connection.maxidletime ipc.client.idlethreshold --> hbase.ipc.client.idlethreshold ipc.client.kill.max --> hbase.ipc.client.kill.max ipc.server.scan.vtime.weight --> hbase.ipc.server.scan.vtime.weight

Description

There is an option to set tcpNoDelay, defaulted to true, but the socket channel is actually not changed. As a consequence, the server works with nagle enabled. This leads to very degraded behavior when a single connection is shared between threads. We enter into conflicts with nagle and tcp delayed ack.

Here is an example of performance with the PE tool plus ~~HBASE-11491~~:

oneCon     #client       sleep          exeTime (seconds)                             avg latency, sleep excluded (microseconds)
true           1               0                31                                                     310
false          1               0                31                                                     310
true           2               0                50                                                      500
false          2               0               31                                                      310
true           2                5               488 (including 200s sleeping)               2880 
false          2               5               246  (including 200s sleeping)               460

The latency is multiple by 5 (2880 vs 460) when the connection is shared. This is the delayed ack kicking in. This can be fixed by really using tcp no delay.

Any application sharing the tcp connection between threads has the issue.

Attachments

- Sort By Name
- Sort By Date
- Ascending
- Descending

11492.v3.patch
21/Jul/14 12:15
13 kB
Nicolas Liochon
11492.v2-0.98.patch
11/Jul/14 20:46
11 kB
Andrew Kyle Purtell
11492.v2.patch
11/Jul/14 17:35
5 kB
Nicolas Liochon
11492.v2.patch
13/Jul/14 20:38
5 kB
Nicolas Liochon
11492.v1.withp1.patch
11/Jul/14 10:23
1 kB
Nicolas Liochon
11492.v1.patch
10/Jul/14 15:37
1 kB
Nicolas Liochon
11492.098.addendum.v2.patch
21/Jul/14 16:57
10 kB
Andrew Kyle Purtell
11492.098.addendum.patch
21/Jul/14 12:14
9 kB
Nicolas Liochon

Issue Links

breaks

HBASE-11698 TestFullLogReconstruction has duplicate config setting

Closed

Sub-Tasks

There are no Sub-Tasks for this issue.

Hadoop configuration overrides some ipc parameters including tcpNoDelay

Details

Description

Attachments

Attachments

Issue Links

Sub-Tasks

Activity

People

Dates