Uploaded image for project: 'Nuxeo Platform'
  1. Nuxeo Platform
  2. NXP-18050

Refactor and finalize TransientStore

    XMLWordPrintable

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 7.4
    • Fix Version/s: 7.10
    • Component/s: TransientStore
    • Impact type:
      API change
    • Upgrade notes:
      Hide

      Removed:

      • AbstractStorageEntry
      • org.nuxeo.ecm.core.transientstore.api.StorageEntry
      • StorageEntryImpl
      • BatchChunkEntry

      Removed from TransientStore:

      • void put(StorageEntry entry)
      • StorageEntry get(String key);

      Removed from TransientStoreWork:

      • void saveStorageEntry(StorageEntry storageEntry)
      • StorageEntry getStorageEntry(String key)
      • void removeStorageEntry(String key)

      Added:

      • org.nuxeo.ecm.core.transientstore.StorageEntry

      Added to TransientStore:

      • boolean exists(String key)
      • void putParameter(String key, String parameter, Serializable value
      • Serializable getParameter(String key, String parameter)
      • void putParameters(String key, Map<String, Serializable> parameters)
      • Map<String, Serializable> getParameters(String key)
      • void putBlobs(String key, List<Blob> blobs)
      • List<Blob> getBlobs(String key)
      • long getSize(String key)
      • boolean isCompleted(String key)
      • void setCompleted(String key, boolean completed)
      • void removeAll()

      Added to TransientStoreWork:

      • void putBlobHolder(String key, BlobHolder bh)
      • BlobHolder getBlobHolder(String key)
      • void removeBlobHolder(String key)
      Show
      Removed: AbstractStorageEntry org.nuxeo.ecm.core.transientstore.api.StorageEntry StorageEntryImpl BatchChunkEntry Removed from TransientStore: void put(StorageEntry entry) StorageEntry get(String key); Removed from TransientStoreWork: void saveStorageEntry(StorageEntry storageEntry) StorageEntry getStorageEntry(String key) void removeStorageEntry(String key) Added: org.nuxeo.ecm.core.transientstore.StorageEntry Added to TransientStore: boolean exists(String key) void putParameter(String key, String parameter, Serializable value Serializable getParameter(String key, String parameter) void putParameters(String key, Map<String, Serializable> parameters) Map<String, Serializable> getParameters(String key) void putBlobs(String key, List<Blob> blobs) List<Blob> getBlobs(String key) long getSize(String key) boolean isCompleted(String key) void setCompleted(String key, boolean completed) void removeAll() Added to TransientStoreWork: void putBlobHolder(String key, BlobHolder bh) BlobHolder getBlobHolder(String key) void removeBlobHolder(String key)
    • Sprint:
      drive-7.10-1, drive-7.10-2
    • Story Points:
      3

      Description

      Main changes to the TransientStore API

      • We are no more manipulating a StorageEntry object to represent what is stored but simply a list of blobs and a map of parameters.
        Thus the new API:
        void putParameters(String key, Map<String, Serializable> parameters);
        
        Map<String, Serializable> getParameters(String key);
        
        void putBlobs(String key, List<Blob> blobs);
        
        List<Blob> getBlobs(String key);
        

        The purpose of these changes is to get closer to the Redis atomic commands to better manage concurrency and atomicity in the Redis implementation.
        Note that a StorageEntry class is still used as a POJO in the SimpleTransientStore in-memory implementation.

      • The CacheService is now only used by the in-memory implementation.
      • The AbstractTransientStore is now only in charge of storing / loading / garbaging the blobs on / from the file system.
        Everything that is related to entry parameters and blob metadata is handled by the implementers.

      This refactoring allows the Redis implementation to be cluster aware, see NXP-18051 for implementation details.

      Batch Upload Example

      As mentioned in NXP-17885, since 7.4 the BatchManager relies on the TransientStore to allow several implementations among which the Redis one that is cluster aware.
      Indeed, upload data (structure and streams) must be shared across Nuxeo nodes if we want the upload system to work across the cluster without having to enforce affinity.

      Here is the new way of storing a batch in the TransientStore, following the refactoring:

      Example of a batch with 2 files among which one of them is made of 2 chunks:

      1. The Batch is stored in the "default" transient store with its id as a key.
        It has no blobs but references in its parameters the keys of each file in the batch, stored in the same transient store as objects represented by the BatchFileEntry class.
        TransientStore("default") -> {"batchId-a0dbccda-a36c-436d-8de6-09fe96f14e08": (
                                          blobs = [],
                                          params = {"0": "batchId-a0dbccda-a36c-436d-8de6-09fe96f14e08_0", "1": "batchId-a0dbccda-a36c-436d-8de6-09fe96f14e08_1"}
        )}
        
      2. Each batch file is indeed stored in the "default" transient store with the file index concatenated to the batch id as a key.
        A file that is not chunked directly references its blob in its blob list, a file that is chunked references in its parameters the keys of each chunk, stored in the same transient store.
        TransientStore("default") -> {"batchId-a0dbccda-a36c-436d-8de6-09fe96f14e08_0": (
                                          blobs = [blob],
                                          params = {"chunked": false}
        )}
        
        TransientStore("default") -> {"batchId-a0dbccda-a36c-436d-8de6-09fe96f14e08_1": (
                                          blobs = [],
                                          params = {"chunked": true, "fileName": "My file.txt", "mimeType": "text/plain", "fileSize": 1024, "chunkCount": 2, "0": "batchId-a0dbccda-a36c-436d-8de6-09fe96f14e08_1_0", 1: "batchId-a0dbccda-a36c-436d-8de6-09fe96f14e08_1_1"}
        )}
        
      3. Each chunk is indeed stored in the "default" transient store with the chunk index concatenated to the file key as a key.
        A chunk directly references its blob in its blob list and has no parameters.
        TransientStore("default") -> {"batchId-a0dbccda-a36c-436d-8de6-09fe96f14e08_1_0": (
                                          blobs = [chunk0],
                                          params = {}
        )}
        
        TransientStore("default") -> {"batchId-a0dbccda-a36c-436d-8de6-09fe96f14e08_1_1": (
                                          blobs = [chunk1],
                                          params = {}
        )}
        

        Attachments

          Issue Links

            Activity

              People

              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: