Uploaded image for project: 'Nuxeo Platform'
  1. Nuxeo Platform
  2. NXP-27881

Check for Kafka topic partitions increase

    XMLWordPrintable

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Minor
    • Resolution: Unresolved
    • Affects Version/s: 10.10
    • Fix Version/s: 2021.x, 2023.x
    • Component/s: Streams

      Description

      The use case is scaling up the number of partitions during processing without having to restart existing nodes.

      The number of partitions is supposed to be a fixed number, still, it is possible to add partitions if the over-provisioning was wrongly estimated.
      Note that:

      • Adding partitions creates disorder, this should not be done if the processing requires ordering.
      • Only the new records will be dispatched to the new partitions
      • There is no way to remove partitions from a topic

      The number of partitions needs to be known by producers to be able to dispatch records according to its key.
      Getting the number of partition on each write will cost too much so this could be done every N records.
      This should be provided as an option (like nuxeo.kafka.check.partitions.records=200 to check every 200 records submitted)

        Attachments

          Activity

            People

            • Votes:
              1 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated: