Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-21517

Fetch local data via block manager cause oom

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 1.6.1, 2.1.0
    • 2.3.0
    • Block Manager, Spark Core
    • None

    Description

      In our production cluster,oom happens when NettyBlockRpcServer receive OpenBlocks message.The reason we observed is below:
      When BlockManagerManagedBuffer call ChunkedByteBuffer#toNetty, it will use Unpooled.wrappedBuffer(ByteBuffer... buffers) which use default maxNumComponents=16 in low-level CompositeByteBuf.When our component's number is bigger than 16, it will execute during buffer copy.

      private void consolidateIfNeeded() {
              int numComponents = this.components.size();
              if(numComponents > this.maxNumComponents) {
                  int capacity = ((CompositeByteBuf.Component)this.components.get(numComponents - 1)).endOffset;
                  ByteBuf consolidated = this.allocBuffer(capacity);
      
                  for(int c = 0; c < numComponents; ++c) {
                      CompositeByteBuf.Component c1 = (CompositeByteBuf.Component)this.components.get(c);
                      ByteBuf b = c1.buf;
                      consolidated.writeBytes(b);
                      c1.freeIfNecessary();
                  }
      
                  CompositeByteBuf.Component var7 = new CompositeByteBuf.Component(consolidated);
                  var7.endOffset = var7.length;
                  this.components.clear();
                  this.components.add(var7);
              }
      
          }
      

      in CompositeByteBuf which will consume some memory during buffer copy.

      Attachments

        Issue Links

          Activity

            People

              caneGuy zhoukang
              cane zhoukang
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: