-
Type: Improvement
-
Status: Resolved
-
Priority: Minor
-
Resolution: Fixed
-
Affects Version/s: None
-
Component/s: Streams
-
Tags:
-
Team:PLATFORM
-
Sprint:nxplatform 11.1.33, nxplatform 11.1.34, nxplatform 11.1.35, nxplatform 11.1.36
-
Story Points:2
In Nuxeo Stream, at the low level, we have a Log storage abstraction accessible using a LogManager,
this LogManager is used to perform basic Producer/Consumer pattern.
A LogManager is configured by LogConfig describing its type (Chronicle or Kafka) and some options like the Kafka configuration to use.
Today there is one LogManager per configuration, and we have multiples LogManagers because we want different configurations between services. There is one for the audit, one for the Bulk Service, one for the StreamWorkManager and a default one.
At a higher level, we have a Stream Processor pattern accessible using a StreamManager,
this manager relies on a LogManager, a Stream being materialized by a Log. Here also multiple StreamManagers.
This multi-managers approach is limiting because:
- A LogManager provides metrics on the async processing, having multiple managers makes it complex to have a global status
- A service uses a single manager and is therefore limited to a single configuration
- Stream Processing cannot be mixed between multiple stream manager
To solve this the idea is to have a single unified LogManager that supports multiple configurations.
The configuration will be applied based on stream and/or consumer group name, using URN for these names as described in NXP-28877 will help here.
This will solve our limitations, providing:
- central access for metrics needed for
NXP-28801 - fine-grained configuration, for example, to apply a specific Kafka configuration for the video processing in the StreamWorkManager
- interoperability between streams:
- use multiple Kafka clusters, a dockerized without HA for mass import in addition to an MSK.
- mix Kafka and CQ within a processing
- depends on
-
NXP-28877 Introduce Namespaces for Stream and Consumer Group names
- Resolved
- is related to
-
NXDOC-2142 Update Nuxeo Stream doc regarding namespace and config
- Resolved
-
NXP-23778 Stream WorkManager must be able to configure LogConfig per worker pool
- Resolved
- is required by
-
NXP-28801 Report Stream lag and latency from Nuxeo
- Resolved