Uploaded image for project: 'Ratis'
  1. Ratis
  2. RATIS-2027

Ratis Streaming: Remote Stream copies data to heap

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 3.1.0
    • Streaming

    Description

      In ratis streaming, the write to RemoteStream uses ByteBuf#nioBuffer() which copies the ByteBuf content to a HeapByteBuffer.

      CompletableFuture<DataStreamReply> write(DataStreamRequestByteBuf request, Executor executor) {
        final Timekeeper.Context context = metrics.start();
        return composeAsync(sendFuture, executor,
            n -> out.writeAsync(request.slice().nioBuffer(), addFlush(request.getWriteOptionList()))
                .whenComplete((l, e) -> metrics.stop(context, e == null)));
      } 

      And this action implies 2 points of inefficiency:

      1. The created and discarded HeapBuffer creates the O(N) cost to GC, with N as the data amount flowing through the ratis server.
      2. When the HeapBuffer is written to the network, it needs to be copied again to a DriectBuffer by NIO. So, in total, 2 copies. 

      See:  dn_write_streaming_allocation.htm and dn_write_streaming.htm

      Attachments

        1. datanode-write-ratis-streaming-after-ratis-2027.png
          985 kB
          Duong
        2. datanode-write-ratis-streaming.png
          935 kB
          Duong
        3. allocation.png
          405 kB
          Duong
        4. cpu.png
          320 kB
          Duong
        5. dn_write_streaming.htm
          436 kB
          Duong
        6. dn_write_streaming_allocation.htm
          1.19 MB
          Duong

        Issue Links

          Activity

            People

              szetszwo Tsz-wo Sze
              duongnguyen Duong
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 50m
                  50m