Uploaded image for project: 'Nuxeo Platform'
  1. Nuxeo Platform
  2. NXP-31311

Create a MongoTransientStore that can handle large number params

    XMLWordPrintable

    Details

    • Release Notes Summary:
      There is a new optimized implementation of TransientStore for MongoDB
    • Tags:
    • Backlog priority:
      600
    • Upgrade notes:
      Hide

      To use a MongoDB optimized implementation of the transientstore, (that is replacing the KeyValueTransientStore implementation), the following property must be se on nuxeo.conf for 10.10 and LTS 2021:

      nuxeo.transientstore.provider=mongodb

      Note that on LTS 2023 this implementation is used by default.

      Show
      To use a MongoDB optimized implementation of the transientstore, (that is replacing the KeyValueTransientStore implementation), the following property must be se on nuxeo.conf for 10.10 and LTS 2021: nuxeo.transientstore.provider=mongodb Note that on LTS 2023 this implementation is used by default.
    • Team:
      PLATFORM
    • Sprint:
      nxplatform #73, nxplatform #74, nxplatform #75, nxplatform #76
    • Story Points:
      8

      Description

      Today the S3DirectBatchHandler backed by a KeyValueTransientStore is generating too many roundtrip with the KVStore we need a better implementation to large batch (more than 5k files)

      S3DirectBatchHandler completeUpload performance degrades when the number of files in the batch increase.

      for instance calling the endoint /nuxeo/api/v1/upload/BATCH_ID/5/complete in a batch of 10 files results in the following kv access:

      GET batchId-12a90393-3b4c-4848-8dd8-79f63b58570e.paraminfo = ["handler","0","1","2","3","4","5","6","8","7","9","10"]
GET batchId-12a90393-3b4c-4848-8dd8-79f63b58570e.param.handler__format = null
GET batchId-12a90393-3b4c-4848-8dd8-79f63b58570e.param.handler = default
GET batchId-12a90393-3b4c-4848-8dd8-79f63b58570e.param.0__format = null
GET batchId-12a90393-3b4c-4848-8dd8-79f63b58570e.param.0 = batchId-12a90393-3b4c-4848-8dd8-79f63b58570e_0
GET batchId-12a90393-3b4c-4848-8dd8-79f63b58570e.param.1__format = null
GET batchId-12a90393-3b4c-4848-8dd8-79f63b58570e.param.1 = batchId-12a90393-3b4c-4848-8dd8-79f63b58570e_1
GET batchId-12a90393-3b4c-4848-8dd8-79f63b58570e.param.2__format = null
GET batchId-12a90393-3b4c-4848-8dd8-79f63b58570e.param.2 = batchId-12a90393-3b4c-4848-8dd8-79f63b58570e_2
GET batchId-12a90393-3b4c-4848-8dd8-79f63b58570e.param.3__format = null
GET batchId-12a90393-3b4c-4848-8dd8-79f63b58570e.param.3 = batchId-12a90393-3b4c-4848-8dd8-79f63b58570e_3
GET batchId-12a90393-3b4c-4848-8dd8-79f63b58570e.param.4__format = null
GET batchId-12a90393-3b4c-4848-8dd8-79f63b58570e.param.4 = batchId-12a90393-3b4c-4848-8dd8-79f63b58570e_4
GET batchId-12a90393-3b4c-4848-8dd8-79f63b58570e.param.5__format = null
GET batchId-12a90393-3b4c-4848-8dd8-79f63b58570e.param.5 = batchId-12a90393-3b4c-4848-8dd8-79f63b58570e_5
GET batchId-12a90393-3b4c-4848-8dd8-79f63b58570e.param.6__format = null
GET batchId-12a90393-3b4c-4848-8dd8-79f63b58570e.param.6 = batchId-12a90393-3b4c-4848-8dd8-79f63b58570e_6
GET batchId-12a90393-3b4c-4848-8dd8-79f63b58570e.param.8__format = null
GET batchId-12a90393-3b4c-4848-8dd8-79f63b58570e.param.8 = batchId-12a90393-3b4c-4848-8dd8-79f63b58570e_8
GET batchId-12a90393-3b4c-4848-8dd8-79f63b58570e.param.7__format = null
GET batchId-12a90393-3b4c-4848-8dd8-79f63b58570e.param.7 = batchId-12a90393-3b4c-4848-8dd8-79f63b58570e_7
GET batchId-12a90393-3b4c-4848-8dd8-79f63b58570e.param.9__format = null
GET batchId-12a90393-3b4c-4848-8dd8-79f63b58570e.param.9 = batchId-12a90393-3b4c-4848-8dd8-79f63b58570e_9
GET batchId-12a90393-3b4c-4848-8dd8-79f63b58570e.param.10__format = null
GET batchId-12a90393-3b4c-4848-8dd8-79f63b58570e.param.10 = batchId-12a90393-3b4c-4848-8dd8-79f63b58570e_10
GET batchId-12a90393-3b4c-4848-8dd8-79f63b58570e.paraminfo = ["handler","0","1","2","3","4","5","6","8","7","9","10"]
GET batchId-12a90393-3b4c-4848-8dd8-79f63b58570e.param.handler__format = null
GET batchId-12a90393-3b4c-4848-8dd8-79f63b58570e.param.handler = default
GET batchId-12a90393-3b4c-4848-8dd8-79f63b58570e.param.0__format = null
GET batchId-12a90393-3b4c-4848-8dd8-79f63b58570e.param.0 = batchId-12a90393-3b4c-4848-8dd8-79f63b58570e_0
GET batchId-12a90393-3b4c-4848-8dd8-79f63b58570e.param.1__format = null
GET batchId-12a90393-3b4c-4848-8dd8-79f63b58570e.param.1 = batchId-12a90393-3b4c-4848-8dd8-79f63b58570e_1
GET batchId-12a90393-3b4c-4848-8dd8-79f63b58570e.param.2__format = null
GET batchId-12a90393-3b4c-4848-8dd8-79f63b58570e.param.2 = batchId-12a90393-3b4c-4848-8dd8-79f63b58570e_2
GET batchId-12a90393-3b4c-4848-8dd8-79f63b58570e.param.3__format = null
GET batchId-12a90393-3b4c-4848-8dd8-79f63b58570e.param.3 = batchId-12a90393-3b4c-4848-8dd8-79f63b58570e_3
GET batchId-12a90393-3b4c-4848-8dd8-79f63b58570e.param.4__format = null
GET batchId-12a90393-3b4c-4848-8dd8-79f63b58570e.param.4 = batchId-12a90393-3b4c-4848-8dd8-79f63b58570e_4
GET batchId-12a90393-3b4c-4848-8dd8-79f63b58570e.param.5__format = null
GET batchId-12a90393-3b4c-4848-8dd8-79f63b58570e.param.5 = batchId-12a90393-3b4c-4848-8dd8-79f63b58570e_5
GET batchId-12a90393-3b4c-4848-8dd8-79f63b58570e.param.6__format = null
GET batchId-12a90393-3b4c-4848-8dd8-79f63b58570e.param.6 = batchId-12a90393-3b4c-4848-8dd8-79f63b58570e_6
GET batchId-12a90393-3b4c-4848-8dd8-79f63b58570e.param.8__format = null
GET batchId-12a90393-3b4c-4848-8dd8-79f63b58570e.param.8 = batchId-12a90393-3b4c-4848-8dd8-79f63b58570e_8
GET batchId-12a90393-3b4c-4848-8dd8-79f63b58570e.param.7__format = null
GET batchId-12a90393-3b4c-4848-8dd8-79f63b58570e.param.7 = batchId-12a90393-3b4c-4848-8dd8-79f63b58570e_7
GET batchId-12a90393-3b4c-4848-8dd8-79f63b58570e.param.9__format = null
GET batchId-12a90393-3b4c-4848-8dd8-79f63b58570e.param.9 = batchId-12a90393-3b4c-4848-8dd8-79f63b58570e_9
GET batchId-12a90393-3b4c-4848-8dd8-79f63b58570e.param.10__format = null
GET batchId-12a90393-3b4c-4848-8dd8-79f63b58570e.param.10 = batchId-12a90393-3b4c-4848-8dd8-79f63b58570e_10
GET batchId-12a90393-3b4c-4848-8dd8-79f63b58570e_5.paraminfo = ["chunked"]
GET batchId-12a90393-3b4c-4848-8dd8-79f63b58570e_5.param.chunked__format = null
GET batchId-12a90393-3b4c-4848-8dd8-79f63b58570e_5.param.chunked = false
TEST batchId-12a90393-3b4c-4848-8dd8-79f63b58570e_5.bloblock = null ? SET true
GET batchId-12a90393-3b4c-4848-8dd8-79f63b58570e_5.blobinfo = {"size":"7680","count":"1"}
GET batchId-12a90393-3b4c-4848-8dd8-79f63b58570e_5.blob.0 = {"filename":"myFile.doc","length":"7680","digest":"858a3dadd76959d5c45e60be671dac53","mimetype":"application/msword","encoding":null,"key":"858a3dadd76959d5c45e60be671dac53"}
DEL batchId-12a90393-3b4c-4848-8dd8-79f63b58570e_5.bloblock

      This is 4xNB_OF_FILES + 10 requests,
      this is not noticeable on a a small batch, each roundtrip being sub millisecond,
      but as soon as there are hundreds of files this is an observed bottleneck.

        Attachments

          Issue Links

          causes

          Bug - A problem which impairs or prevents the functions of the product. NXP-31636 Fix CSV export with MongoDB transient store

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. NXP-31171 AsperaCompletionWork takes too long when number of files increase

          • Major - Major loss of function.
          • Resolved

            Activity

              People

              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: