Uploaded image for project: 'James Server'
  1. James Server
  2. JAMES-2921

Improve blob store usage

    XMLWordPrintableJSON

Details

    • Wish
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • None
    • Blob
    • None

    Description

      Please find attached the two technical decisions being proposed.

      These needs performance feedback before being adopted.

      Proposal: Add storage policies for BlobStore

      Introduce StoragePolicies at the level of the BlobStore API.

      The proposed policies includes:

      • SizeBasedStoragePolicy: The blob underlying storage medium will be chosen depending of its size.
      • LowCostStoragePolicy: The blob is expected to be saved in low cost storage. Access is expected to be unfrequent.
      • PerformantStoragePolicy: The blob is expected to be saved in performant storage. Access is expected to be unfrequent.

      The UnionBlobStore will be reworked to choose between Cassandra and ObjectStorage implementations depending on the policies.

      DeletedMessageVault, BlobExport & MailRepository will rely on LowCostStoragePolicy. Other BlobStore users will rely on SizeBasedStoragePolicy.

      Proposal: persist BlobIds for avoiding persisting several time the same blobs within ObjectStorage

      Rely on a StoredBlobIdsList API to know which blob is persisted or not in object storage. Provide a Cassandra implementation of it. Located in blob-api for convenience, this in not a top level API. It is intended to be used by some blobStore implementations (here only ObjectStorage).

      • When saving a blob with precomputed blobId, we can check the existance of the blob in storage, avoiding possibly the expensive "save".
      • When saving a blob too big to precompute its blobId, once the blob had been stream using a temporary random blobId, copy operation can be avoided and the temporary blob could be directly removed.

      Cassandra is faster doing "write every time" rather than "read before write" so we should not use the storedblob projection for it

      Attachments

        1. adr-0004-objectstorage-blobid-list.md
          2 kB
          Benoit Tellier
        2. adr-0004-objectstorage-blobid-list.md
          2 kB
          Gautier DI FOLCO
        3. adr-0003-blobstore-storage-policies.md
          2 kB
          Benoit Tellier
        4. adr-0003-blobstore-storage-policies.md
          2 kB
          Gautier DI FOLCO

        Activity

          People

            Unassigned Unassigned
            btellier Benoit Tellier
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: