Since NXP-23757 we can reset the consumer positions of a stream. On start, processors (consumers) read from the beginning of stream instead of the last committed position.
We need to extend the reset command so it can take a timestamp. On start, processors will read from the offset corresponding to a message with a greater or equal timestamp.
This option is interesting for multi-DC deployment where Kafka cluster is replicated with mirror maker.
In this case we cannot propagate the consumer offsets between DC, because record offsets are not identical between clusters.
A solution for cluster failover described in KIP-33 is to start from a timestamp which can be: CrashTime - MaxLatencyOfCluster.
This will creates duplicates but this is expected in case of failure and workers should handle this.
Nuxeo don't set the timestamp of Kafka message which mean that the message timestamp is set by the Kafka broker as LogAppendTime, so we should be able to find offset from timestamp.
It may be not possible to do the same with Chronicle (may be the timestamp can be decoded from the offset).
- is related to
-
NXP-24467 TestLogKafka.testSeekByTimestamp intermittent failures
- Resolved
-
NXP-24156 Extend the stream.sh to set consumer position to a watermark
- Resolved
-
NXP-23757 Make Stream tool work with kafka
- Resolved
- is required by
-
NXP-24197 Workmanager should replay only last replicated works
- In Progress