Uploaded image for project: 'Nuxeo Platform'
  1. Nuxeo Platform
  2. NXP-18051

Refactor RedisTransientStore to have a working cluster aware TransientStore

    XMLWordPrintable

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 7.4
    • Fix Version/s: 7.10
    • Component/s: Redis, TransientStore

      Description

      StorageEntry storage: previous state

      Previously the StorageEntry was stored inside 2 Caches depending on their lifecycle.

      Because it was using the Cache API, the StorageEntry was stored as a whole as a single serialized Java object.

      So we had something like:

      cache:storageId1: <serializeddata>
      cache:storageId2: <serializeddata>
      

      This was creating several issues:

      • Double init of an entry and dirty updates
      • Back & forth serialization
      • Poor lisibility

      Improvements

      We have reviewed the RedisTransientStore to bypass the Cache usage.

      The idea was to use a native Redis data structure that:

      • Manages concurrency and atomicity.
      • Limits serialization usage.

      This fixes the concurrency issue faced in the BatchManager and previously handled by synchronized blocks.

      From a JSON perspective a storage entry can be seen as:

      {
          “blobs” : [{…}, {…}, ],
          “params” : {…},
          “completed” : false,
          “size” : 123
      }
      

      We wanted to keep this structure inside Redis, but since we could not nest the hashes, we needed to flatten it.

      # Store the main map
      > hmset ts:storeId:entryKey blobCount 2 completed false size 123
      
      ts:storeId:entryKey    {
                                       "blobCount" : 2,
                                       "completed" : false,
                                       "size" : 123
                                      }
      
      # Store the params map
      > hmset ts:storeId:entryKey:params  p1 "value1" p2 "java:serializeddata"
      
      ts:storeId:entryKey:params    {
                                               "p1" : "value1",
                                               "p2" : "java:serializeddata"
                                              }
      # Store the first blob
      > hmset ts:storeId:entryKey:blobs:0 file "relative/path/to/file" filename "xxx" mimetype "yyy" encoding "zzz" digest "ddd"
      
      ts:storeId:entryKey:blobs:0     {
                                                      "file" : "relative/path/to/file"
                                                      "filename" : "xxx",
                                                      "mimetype" : "yyy",
                                                      "encoding" : "zzz",
                                                      "digest" : "ddd",
                                                    }
      
      # Store the second blob
      ...
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                ataillefer Antoine Taillefer
                Reporter:
                ataillefer Antoine Taillefer
                Participants:
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  PagerDuty

                  Error rendering 'com.pagerduty.jira-server-plugin:PagerDuty'. Please contact your Jira administrators.