-
Type: Improvement
-
Status: Resolved
-
Priority: Major
-
Resolution: Fixed
-
Affects Version/s: 7.4
-
Fix Version/s: 7.10
-
Component/s: TransientStore
-
Tags:
-
Impact type:API change
-
Upgrade notes:
-
Sprint:drive-7.10-1, drive-7.10-2
-
Story Points:3
Main changes to the TransientStore API
- We are no more manipulating a StorageEntry object to represent what is stored but simply a list of blobs and a map of parameters.
Thus the new API:void putParameters(String key, Map<String, Serializable> parameters); Map<String, Serializable> getParameters(String key); void putBlobs(String key, List<Blob> blobs); List<Blob> getBlobs(String key);
The purpose of these changes is to get closer to the Redis atomic commands to better manage concurrency and atomicity in the Redis implementation.
Note that a StorageEntry class is still used as a POJO in the SimpleTransientStore in-memory implementation.
- The CacheService is now only used by the in-memory implementation.
- The AbstractTransientStore is now only in charge of storing / loading / garbaging the blobs on / from the file system.
Everything that is related to entry parameters and blob metadata is handled by the implementers.
This refactoring allows the Redis implementation to be cluster aware, see NXP-18051 for implementation details.
Batch Upload Example
As mentioned in NXP-17885, since 7.4 the BatchManager relies on the TransientStore to allow several implementations among which the Redis one that is cluster aware.
Indeed, upload data (structure and streams) must be shared across Nuxeo nodes if we want the upload system to work across the cluster without having to enforce affinity.
Here is the new way of storing a batch in the TransientStore, following the refactoring:
Example of a batch with 2 files among which one of them is made of 2 chunks:
- The Batch is stored in the "default" transient store with its id as a key.
It has no blobs but references in its parameters the keys of each file in the batch, stored in the same transient store as objects represented by the BatchFileEntry class.TransientStore("default") -> {"batchId-a0dbccda-a36c-436d-8de6-09fe96f14e08": ( blobs = [], params = {"0": "batchId-a0dbccda-a36c-436d-8de6-09fe96f14e08_0", "1": "batchId-a0dbccda-a36c-436d-8de6-09fe96f14e08_1"} )}
- Each batch file is indeed stored in the "default" transient store with the file index concatenated to the batch id as a key.
A file that is not chunked directly references its blob in its blob list, a file that is chunked references in its parameters the keys of each chunk, stored in the same transient store.TransientStore("default") -> {"batchId-a0dbccda-a36c-436d-8de6-09fe96f14e08_0": ( blobs = [blob], params = {"chunked": false} )} TransientStore("default") -> {"batchId-a0dbccda-a36c-436d-8de6-09fe96f14e08_1": ( blobs = [], params = {"chunked": true, "fileName": "My file.txt", "mimeType": "text/plain", "fileSize": 1024, "chunkCount": 2, "0": "batchId-a0dbccda-a36c-436d-8de6-09fe96f14e08_1_0", 1: "batchId-a0dbccda-a36c-436d-8de6-09fe96f14e08_1_1"} )}
- Each chunk is indeed stored in the "default" transient store with the chunk index concatenated to the file key as a key.
A chunk directly references its blob in its blob list and has no parameters.TransientStore("default") -> {"batchId-a0dbccda-a36c-436d-8de6-09fe96f14e08_1_0": ( blobs = [chunk0], params = {} )} TransientStore("default") -> {"batchId-a0dbccda-a36c-436d-8de6-09fe96f14e08_1_1": ( blobs = [chunk1], params = {} )}
- depends on
-
NXP-16577 Provide a service to handle transient storage
- Resolved
-
NXP-18238 Fix typo in property name TransientStoreConfig.fistLevelTTL
- Resolved
-
NXP-17884 Improve TransientStore API
- Resolved
- is required by
-
NXP-17780 Make Upload API Cluster Aware
- Resolved
-
NXP-18236 Use transactions when required in RedisTransientStore
- Open
-
NXP-18244 Handle TTL in RedisTransientStore
- Resolved
-
NXDOC-683 Add doc about the TransientStore
- Resolved
-
NXP-18034 Remove @Experimental from TransientStore API
- Resolved
-
NXP-18187 Run BatchManager and BatchUpload unit tests against a Redis implementation of the TransientStore
- Resolved
-
NXP-18197 Add multithreaded unit tests to validate the BatchManager thread safety
- Resolved
-
NXP-18246 A few TransientStore improvements
- Open
-
NXP-18051 Refactor RedisTransientStore to have a working cluster aware TransientStore
- Resolved
-
NXP-18052 Migrate the Converter cache to the CacheService / TransientStore
- Resolved
-
NXP-18245 Migrate ad hoc blob caches to the TransientStore
- Resolved