-
Type: Improvement
-
Status: Resolved
-
Priority: Minor
-
Resolution: Fixed
-
Affects Version/s: None
-
Component/s: Streams
-
Tags:
-
Upgrade notes:
-
Team:PLATFORM
-
Sprint:nxplatform 11.1.33, nxplatform 11.1.34, nxplatform 11.1.35, nxplatform 11.1.36
-
Story Points:0
Today a Stream (Nuxeo Stream) is designated by a String, for instance `default` is the stream name for the StreamWorkManager `default` queue.
This stream name is translated into a topic name in Kafka using a configurable prefix, in our case:
nuxeo-default
This ensures that we can share a Kafka cluster between multiple Nuxeo instances, achieving multi-tenancy at the Kafka level.
When using Chronicle instead of Kafka, the stream name is translated into a directory:
${NUXEO_DATA}/stream/work/default
The stream name is also used in metric names like
# graphite servers.nuxeo-node-1.nuxeo.nuxeo.streams.computation.default.failure.count # prometheus dropwizard5_nuxeo_streams_computation_failure_counter{computation="default",instance="nuxeo-node-1",job="nuxeo"}
Within a single Nuxeo instance, the number of streams is increasing and conflict may happen between services.
The idea is to use a more structured stream name that includes a namespace and that can be represented as a URN (Unified Resource Name):
namespace/specificName
And also encoded as an identifier when used as a topic, folder or metric name:
namespace-specificName
For instance the StreamWorkManger's default stream will be renamed into work/default, preventing conflict with other services that want to use a default stream.
The same problem exists with the consumer group name, in Kafka we don't want any conflict with this identifier within a Kafka cluster, so it is already prefixed but we also want to add this URN name to avoid conflict within a Nuxeo instance.
Note that the goal is to avoid conflicts and not to achieve isolation between namespace.
Also, this will add meaning to the stream and consumer group's name, giving a better understanding and operability at the Kafka management level,
this is very important when the number of topics/consumer increases or when the Kafka cluster is shared with other applications.
The consequence of the upgrade is a renaming of topics used by Nuxeo, the upgrade will require to wait for async processing, then stop the cluster a drop of existing topics.
- is related to
-
AICORE-301 Rename Nuxeo Stream to use dedicated namespace
- Resolved
-
NXP-28801 Report Stream lag and latency from Nuxeo
- Resolved
- is required by
-
NXP-28879 Provide an Unified Stream/Log Manager
- Resolved
- Is referenced in