Uploaded image for project: 'James Server'
  1. James Server
  2. JAMES-3555

Allow 'real' streaming upon BlobStore reads

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Closed
    • Major
    • Resolution: Won't Fix
    • 3.6.0
    • None
    • Blob
    • None

    Description

      I investigated a build instability... https://github.com/apache/james-project/pull/370

      One time out of 25 the uploaded content was altered (Different SHA 256 in
      the data store). Several Apache CI builds had been failing because of this.

      Investigating this, I found that a piped input stream strategy was not
      hitting this pitfall (RectorUtils::toInputStream). I thus incriminate
      some weird data races on the Reactor side...

      Other areas of the code are likely affected as both the Cassandra and the
      S3 blobStores relies on those for streaming (only relied upon for
      attachment upload).

      Changing the BlobStore API should likely be considered:

      public interface BlobStore {
          Flux<ByteBuffer> read(BucketName bucketName, BlobId blobId);
      }
      

      Why:

      • Supported by both implementation (Cassandra S3)
      • Avoids intermediate transformations requiring blocking
        operations/dedicated thread resources
      • We convert a Flux[ByteBuffer] => InputStream (to conform to BlobStore API) => Flux[ByteBuffer] (reactor-netty layer)

      Attachments

        Activity

          People

            Unassigned Unassigned
            btellier Benoit Tellier
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: