Uploaded image for project: 'Nuxeo Platform'
  1. Nuxeo Platform
  2. NXP-24319

Dedicated background/slow work queues

    XMLWordPrintable

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Minor
    • Resolution: Won't Fix
    • Affects Version/s: 9.10
    • Fix Version/s: None
    • Component/s: Events / Works, Streams

      Description

      Today we can have

      • dedicated Nuxeo nodes that process only work from specific queues, this is useful to scale and to separate different type of load.
      • dedicated Nuxeo nodes that don't process any work, this can be useful to give high UI responsiveness.
      • dedicated group of Nuxeo nodes that use different work queues. For instance we can have a group fo Web nodes that process only work comming from the UI, while another group of nodes is processing works from a mass import. Both groups run the same type of processing but using different work queues.This way user don't have to wait for the end of a mass import to have feedback on their new document.

      What is missing is to be able within a node to use different work queues so a work can be routed depending on its content.

      The use case is to have dedicate queues for slow work, so we can handle slow work (like a work taking 1h for video processing) without blocking others.
      This is also necessary because slow work is problematic for Kafka, it forces to use high poll interval which impact the time of failure detection.

      The choice of the queue can be done at the scheduling if we can predict the duration (using the size of the blob).
      Another solution is to have a processing work timeout, if the timeout is reach the work is stopped, moved to another dedicated queue and we continue processing works.

      Also a consequence is that these slow work can not be ordered.

        Attachments

          Issue Links

            Activity

              People

              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: