-
Type: Improvement
-
Status: Resolved
-
Priority: Minor
-
Resolution: Fixed
-
Affects Version/s: None
-
Component/s: Streams
-
Release Notes Summary:Stream.sh supports the tracking of consumer latency.
-
Epic Link:
-
Sprint:nxcore 10.1.5, nxcore 10.1.6
-
Story Points:3
The idea is to have a command to track periodically the latencies of consumers.
This latencies information can be replicated in a different datacenter and used to position the consumers in case of failover.
The latencies information are send into a Log, (for instance it can be named _consumer_latencies)
The latency is expressed
- with a key composed of:
- the consumer group
- log name
- partition
- and a value composed of:
- the date where the latency has been measured
- the timestamp of the last processed record
- eventually the key of the last processed record
This way another command can use this log to restore the consumer positions at a given time.
The rational for the composed key is to use the Kafka log compaction feature so the _consumer_latencies log can stay compact over time and the command to restore the position can quickly get the latest latency.