Uploaded image for project: 'Nuxeo Platform'
  1. Nuxeo Platform
  2. NXP-28879

Provide an Unified Stream/Log Manager

    XMLWordPrintable

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 11.1, 2021.0
    • Component/s: Streams
    • Tags:
    • Team:
      PLATFORM
    • Sprint:
      nxplatform 11.1.33, nxplatform 11.1.34, nxplatform 11.1.35, nxplatform 11.1.36
    • Story Points:
      2

      Description

      In Nuxeo Stream, at the low level, we have a Log storage abstraction accessible using a LogManager,
      this LogManager is used to perform basic Producer/Consumer pattern.

      A LogManager is configured by LogConfig describing its type (Chronicle or Kafka) and some options like the Kafka configuration to use.

      Today there is one LogManager per configuration, and we have multiples LogManagers because we want different configurations between services. There is one for the audit, one for the Bulk Service, one for the StreamWorkManager and a default one.

      At a higher level, we have a Stream Processor pattern accessible using a StreamManager,
      this manager relies on a LogManager, a Stream being materialized by a Log. Here also multiple StreamManagers.

      This multi-managers approach is limiting because:

      • A LogManager provides metrics on the async processing, having multiple managers makes it complex to have a global status
      • A service uses a single manager and is therefore limited to a single configuration
      • Stream Processing cannot be mixed between multiple stream manager

      To solve this the idea is to have a single unified LogManager that supports multiple configurations.

      The configuration will be applied based on stream and/or consumer group name, using URN for these names as described in NXP-28877 will help here.

      This will solve our limitations, providing:

      • central access for metrics needed for NXP-28801
      • fine-grained configuration, for example, to apply a specific Kafka configuration for the video processing in the StreamWorkManager
      • interoperability between streams:
        • use multiple Kafka clusters, a dockerized without HA for mass import in addition to an MSK.
        • mix Kafka and CQ within a processing

       

        Attachments

          Issue Links

            Activity

              People

              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0 minutes
                  0m
                  Logged:
                  Time Spent - 1 day
                  1d