Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-13799

Improve documentation for Kafka zero-copy

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 3.3.0
    • documentation
    • None

    Description

      Via documentation https://kafka.apache.org/documentation/#maximizingefficiency
      and https://kafka.apache.org/documentation/#networklayer ,
      We can know that Kafka combines pagecache and zero-copy when reading messages in files on disk, which greatly improves the consumption rate of messages.
      But after browsing the source code:
      Look directly at the FileRecords.writeTo(...) method,
      1. Only PlaintextTransportLayer.transferFrom() uses fileChannel.transferTo(), and the bottom layer calls the sendfile method to implement zero-copy data transfer.
      2. The logic of the SslTransportLayer.transferFrom() method: 

      fileChannel.read(fileChannelBuffer, pos) 
      -> 
      sslEngine.wrap(src, netWriteBuffer) 
      -> 
      flush(ByteBuffer buf) && socketChannel.write(buf)

      That is, first read the data on the disk or directly from the page cache, then encrypt the data, and finally send the encrypted data to the network. FileChannel.transferTo() is not used in the whole process.

       

      Conclusion: 

      PlaintextTransportLayer and SslTransportLayer both use pagecache, but SslTransportLayer does not implement zero-copy.

      Attachments

        Issue Links

          Activity

            People

              RivenSun RivenSun
              RivenSun RivenSun
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: