Uploaded image for project: 'Nuxeo Platform'
  1. Nuxeo Platform
  2. NXP-23954

Extend the stream.sh script to set consumer position to a timestamp or end of a Log

    XMLWordPrintable

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 9.10
    • Fix Version/s: 9.10-HF02, 10.1
    • Component/s: Streams

      Description

      Since NXP-23757 we can reset the consumer positions of a stream. On start, processors (consumers) read from the beginning of stream instead of the last committed position.

      We need to extend the reset command so it can take a timestamp. On start, processors will read from the offset corresponding to a message with a greater or equal timestamp.

      This option is interesting for multi-DC deployment where Kafka cluster is replicated with mirror maker.

      In this case we cannot propagate the consumer offsets between DC, because record offsets are not identical between clusters.

      A solution for cluster failover described in KIP-33 is to start from a timestamp which can be: CrashTime - MaxLatencyOfCluster.

      This will creates duplicates but this is expected in case of failure and workers should handle this.

      Nuxeo don't set the timestamp of Kafka message which mean that the message timestamp is set by the Kafka broker as LogAppendTime, so we should be able to find offset from timestamp.

      It may be not possible to do the same with Chronicle (may be the timestamp can be decoded from the offset).

        Attachments

          Issue Links

            Activity

              People

              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - 0 minutes
                  0m
                  Remaining:
                  Remaining Estimate - 0 minutes
                  0m
                  Logged:
                  Time Spent - 3 days, 5 hours
                  3d 5h