Uploaded image for project: 'Ratis'
  1. Ratis
  2. RATIS-1938

Optimize stream friendly small file transfers

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Open
    • Major
    • Resolution: Unresolved
    • None
    • None
    • Streaming
    • None

    Description

      Currently, stream writing requires 3 steps
      1. stream init
      2. write data
      3. close stream

      For small files can we optimize this transfer process by combining these three steps into one and using a single rpc to merge the three steps?

       

      message DataStreamPacketHeaderProto {
        enum Type {
          STREAM_HEADER = 0;
          STREAM_DATA = 1;
          STREAM_HEADER_DATA = 2;  // Add a new header, the init packet contains the data and is transmitted and then closed.   
        }
      
        enum Option {
          SYNC = 0;
          CLOSE = 1;
        }
      
        bytes clientId = 1;
        Type type = 2;
        uint64 streamId = 3;
        uint64 streamOffset = 4;
        uint64 dataLength = 5;
        repeated Option options = 6;
      } 

       

      I understand that https://issues.apache.org/jira/browse/RATIS-1157 can optimize the current process, but for small files it still keeps stream init.

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              Nicholas Niu GuoHao
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated: