Uploaded image for project: 'Nuxeo Platform'
  1. Nuxeo Platform
  2. NXP-28877

Introduce Namespaces for Stream and Consumer Group names

    XMLWordPrintable

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 11.1
    • Component/s: Streams
    • Tags:
    • Upgrade notes:
      Hide

      jjStream and Consumer Group names have changed.
      To upgrade:
      1. Wait for any asynchronous processing to be done, there must be no activity and no error

      2. Stop the Nuxeo cluster

      3. Drop the entire Kafka topics or Chronicle Queue directory

      4. Upgrade the Nuxeo cluster, streams will be created with new names

       

      Note that custom code is backward compatible and will continue to work even with name without namespace, but names have been enforced and must match:

      [A-Za-z0-9][A-Za-z0-9_]*

      or you get:

       IllegalArgumentException: Invalid name without namespace: 'foo.bar'
      Show
      jjStream and Consumer Group names have changed. To upgrade: 1. Wait for any asynchronous processing to be done, there must be no activity and no error 2. Stop the Nuxeo cluster 3. Drop the entire Kafka topics or Chronicle Queue directory 4. Upgrade the Nuxeo cluster, streams will be created with new names   Note that custom code is backward compatible and will continue to work even with name without namespace, but names have been enforced and must match: [A-Za-z0-9] [A-Za-z0-9_] * or you get: IllegalArgumentException: Invalid name without namespace: 'foo.bar'
    • Team:
      PLATFORM
    • Sprint:
      nxplatform 11.1.33, nxplatform 11.1.34, nxplatform 11.1.35, nxplatform 11.1.36
    • Story Points:
      0

      Description

      Today a Stream (Nuxeo Stream) is designated by a String, for instance `default` is the stream name for the StreamWorkManager `default` queue.

      This stream name is translated into a topic name in Kafka using a configurable prefix, in our case:

      nuxeo-default

      This ensures that we can share a Kafka cluster between multiple Nuxeo instances, achieving multi-tenancy at the Kafka level.

      When using Chronicle instead of Kafka, the stream name is translated into a directory:

      ${NUXEO_DATA}/stream/work/default 

      The stream name is also used in metric names like

      # graphite
      servers.nuxeo-node-1.nuxeo.nuxeo.streams.computation.default.failure.count
      # prometheus
      dropwizard5_nuxeo_streams_computation_failure_counter{computation="default",instance="nuxeo-node-1",job="nuxeo"}

      Within a single Nuxeo instance, the number of streams is increasing and conflict may happen between services.

      The idea is to use a more structured stream name that includes a namespace and that can be represented as a URN (Unified Resource Name):

      namespace/specificName

      And also encoded as an identifier when used as a topic, folder or metric name:

      namespace-specificName 

      For instance the StreamWorkManger's default stream will be renamed into work/default, preventing conflict with other services that want to use a default stream.

      The same problem exists with the consumer group name, in Kafka we don't want any conflict with this identifier within a Kafka cluster, so it is already prefixed but we also want to add this URN name to avoid conflict within a Nuxeo instance.

      Note that the goal is to avoid conflicts and not to achieve isolation between namespace.

      Also, this will add meaning to the stream and consumer group's name, giving a better understanding and operability at the Kafka management level,
      this is very important when the number of topics/consumer increases or when the Kafka cluster is shared with other applications.

       

      The consequence of the upgrade is a renaming of topics used by Nuxeo, the upgrade will require to wait for async processing, then stop the cluster a drop of existing topics.

       

        Attachments

          Issue Links

            Activity

              People

              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - 0 minutes
                  0m
                  Remaining:
                  Remaining Estimate - 0 minutes
                  0m
                  Logged:
                  Time Spent - 1 day
                  1d

                    PagerDuty

                    Error rendering 'com.pagerduty.jira-server-plugin:PagerDuty'. Please contact your Jira administrators.