In Nuxeo Stream, at the low level, we have a Log storage abstraction accessible using a LogManager,
this LogManager is used to perform basic Producer/Consumer pattern.
A LogManager is configured by LogConfig describing its type (Chronicle or Kafka) and some options like the Kafka configuration to use.
Today there is one LogManager per configuration, and we have multiples LogManagers because we want different configurations between services. There is one for the audit, one for the Bulk Service, one for the StreamWorkManager and a default one.
At a higher level, we have a Stream Processor pattern accessible using a StreamManager,
this manager relies on a LogManager, a Stream being materialized by a Log. Here also multiple StreamManagers.
This multi-managers approach is limiting because:
- A LogManager provides metrics on the async processing, having multiple managers makes it complex to have a global status
- A service uses a single manager and is therefore limited to a single configuration
- Stream Processing cannot be mixed between multiple stream manager
To solve this the idea is to have a single unified LogManager that supports multiple configurations.
The configuration will be applied based on stream and/or consumer group name, using URN for these names as described in
NXP-28877 will help here.
This will solve our limitations, providing:
- central access for metrics needed for
- fine-grained configuration, for example, to apply a specific Kafka configuration for the video processing in the StreamWorkManager
- interoperability between streams:
- use multiple Kafka clusters, a dockerized without HA for mass import in addition to an MSK.
- mix Kafka and CQ within a processing