Uploaded image for project: 'Nuxeo Platform'
  1. Nuxeo Platform
  2. NXP-21459

Create a new addon for producers/consumers with multi queues



    • Type: Improvement
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 8.10-HF01, 9.1
    • Component/s: Importer


      There are lot of work not merged to master on nuxeo-importer-queue related to:

      • consumer/producer threadpool management
      • message marshalling
      • random Nuxeo document producers
      • consumer offset persistence

      Merging these work will require some change on the legacy nuxeo-platform-importer module, for instance the SourceNode is too tied with the way to browse the documents (a node has to know how to list its children). This extra complexity is not needed and not wanted to perform high volume import.

      It is easier to create a new addon (nuxeo-mqueues) which offer a generic producers/consumers of messages relying on a set of unbounded persisted queues (multi queues) without dependencies on the nuxeo-platform-importer addon.

      This addon will deprecate the nuxeo-importer-queue which is ok since this module is marked as experimental and only used by one known customer.

      The new addon must provide some producer/consumer implementations for Nuxeo document:

      • a random Nuxeo document producer that generates document message the same way as the random importer in nuxeo-platform-importer.
      • a Nuxeo document consumer that create Nuxeo document

      Also related to NXBT-1340 we can use this pattern to have producer/consumer for Blob:

      • producer generates some random text blob
      • consumer upload the blob into a Nuxeo binary store.

      Chaining the blob import and document import will enable to perform NXBT-1340 task.

      The consumer/producer runner should be exposed by automation operation.


          Issue Links



              • Votes:
                0 Vote for this issue
                4 Start watching this issue


                • Created: