Details
-
Improvement
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
1.6.1, 2.1.0
-
None
Description
In our production cluster,oom happens when NettyBlockRpcServer receive OpenBlocks message.The reason we observed is below:
When BlockManagerManagedBuffer call ChunkedByteBuffer#toNetty, it will use Unpooled.wrappedBuffer(ByteBuffer... buffers) which use default maxNumComponents=16 in low-level CompositeByteBuf.When our component's number is bigger than 16, it will execute during buffer copy.
private void consolidateIfNeeded() { int numComponents = this.components.size(); if(numComponents > this.maxNumComponents) { int capacity = ((CompositeByteBuf.Component)this.components.get(numComponents - 1)).endOffset; ByteBuf consolidated = this.allocBuffer(capacity); for(int c = 0; c < numComponents; ++c) { CompositeByteBuf.Component c1 = (CompositeByteBuf.Component)this.components.get(c); ByteBuf b = c1.buf; consolidated.writeBytes(b); c1.freeIfNecessary(); } CompositeByteBuf.Component var7 = new CompositeByteBuf.Component(consolidated); var7.endOffset = var7.length; this.components.clear(); this.components.add(var7); } }
in CompositeByteBuf which will consume some memory during buffer copy.
Attachments
Issue Links
- is related to
-
SPARK-25055 MessageWithHeader transfer ByteBuffer from Netty's CompositeByteBuf many times
- Resolved
- links to