-
Type: Improvement
-
Status: Open
-
Priority: Minor
-
Resolution: Unresolved
-
Affects Version/s: 10.10
-
Component/s: Streams
-
Epic Link:
-
Tags:
-
Story Points:3
The use case is scaling up the number of partitions during processing without having to restart existing nodes.
The number of partitions is supposed to be a fixed number, still, it is possible to add partitions if the over-provisioning was wrongly estimated.
Note that:
- Adding partitions creates disorder, this should not be done if the processing requires ordering.
- Only the new records will be dispatched to the new partitions
- There is no way to remove partitions from a topic
The number of partitions needs to be known by producers to be able to dispatch records according to its key.
Getting the number of partition on each write will cost too much so this could be done every N records.
This should be provided as an option (like nuxeo.kafka.check.partitions.records=200 to check every 200 records submitted)