Uploaded image for project: 'Nuxeo Platform'
  1. Nuxeo Platform
  2. NXP-29208

Fix Kafka duplicate processing due to Incomplete rebalance

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2021.18
    • Component/s: Streams
    • Release Notes Summary:
      Avoid duplicate processing when Kafka rebalancing is incomplete
    • Team:
      PLATFORM
    • Sprint:
      nxplatform #57
    • Story Points:
      3

      Description

      Some unit tests are failing in CI using Kafka, each time there is a WARN about "Incomplete rebalance during poll", the assertion is always failing because there are too much results probably because of duplicate processing:

      org.nuxeo.lib.stream.tests.computation.TestLogStreamProcessorKafka.testComplexTopoFewRecords
      Failing for the past 1 build (Since Failed#510 )
      Took 3.2 sec.
      add description
      Error Message
      
      expected:<136> but was:<226>
      
      Stacktrace
      
      java.lang.AssertionError: expected:<136> but was:<226>
      	at org.junit.Assert.fail(Assert.java:89)
      	at org.junit.Assert.failNotEquals(Assert.java:835)
      	at org.junit.Assert.assertEquals(Assert.java:647)
      	at org.junit.Assert.assertEquals(Assert.java:633)
      	at org.nuxeo.lib.stream.tests.computation.TestStreamProcessor.testComplexTopo(TestStreamProcessor.java:251)
      	at org.nuxeo.lib.stream.tests.computation.TestStreamProcessor.testComplexTopoFewRecords(TestStreamProcessor.java:282)
      ...
      Standard Output
      
      09:04:09,635 [COUNTERPool-02,in:91,inCheckpoint:1,out:3,lastRead:1591175049634,lastTimer:1591175049604,wm:208558495937724417,loop:96,rebalance revoked] WARN  [KafkaLogTailer] Incomplete rebalance during poll, raising exception, revoked: true, lost: false
      09:04:09,747 [C3Pool-00,in:16,inCheckpoint:16,out:16,lastRead:1591175049481,lastTimer:0,wm:208558495939428353,loop:21,rebalance revoked] WARN  [KafkaLogTailer] Incomplete rebalance during poll, raising exception, revoked: true, lost: false
      09:04:09,843 [C3Pool-02,in:8,inCheckpoint:8,out:8,lastRead:1591175049464,lastTimer:0,wm:208558495939166209,loop:14,rebalance revoked] WARN  [KafkaLogTailer] Incomplete rebalance during poll, raising exception, revoked: true, lost: false
      09:04:09,845 [C3Pool-01,in:10,inCheckpoint:10,out:10,lastRead:1591175049465,lastTimer:0,wm:208558495939035137,loop:16,rebalance revoked] WARN  [KafkaLogTailer] Incomplete rebalance during poll, raising exception, revoked: true, lost: false
      

        Attachments

          Issue Links

            Activity

              People

              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  PagerDuty

                  Error rendering 'com.pagerduty.jira-server-plugin:PagerDuty'. Please contact your Jira administrators.