Uploaded image for project: 'Nuxeo Platform'
  1. Nuxeo Platform
  2. NXP-27471

Expose stream processor failures as metrics

    XMLWordPrintable

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 10.10
    • Fix Version/s: 11.1, 2021.0
    • Component/s: Streams

      Description

      Since NXP-27164 there is a probe to report stream processor failure. Because probes are used through the runningstatus as health status, the result is that a record that creates a systematic error on the processing will block the entire system.
      This can be mitigated by using proper retry policy for a temporary failure (service unavailable or in failure that requires human intervention) but this is problematic for a buggy record that creates a systematic error.
      So instead of activating a probe for the stream processor, we could have metrics on processing in error that can be used as a warning in a monitoring dashboard.
      This way the ops can choose when to restart Nuxeo node instead of having them automatically blacklisted or restarted.

      The solution is to expose a counter metric when the processing enters in termination due to error, also even if the probe is disabled it will be nice to have the stream processor probe output to list which processing is failing.

        Attachments

        1. Grafana Stream failure counter.png
          3 kB
          Benoit Delbosc
        2. screenshot-1.png
          17 kB
          Benoit Delbosc
        3. screenshot-2.png
          21 kB
          Benoit Delbosc
        4. screenshot-3.png
          30 kB
          Benoit Delbosc
        5. screenshot-4.png
          45 kB
          Benoit Delbosc

          Issue Links

          depends on

          Bug - A problem which impairs or prevents the functions of the product. NXP-27525 Disabling healthCheck probe does not work

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved
          is related to

          Bug - A problem which impairs or prevents the functions of the product. NXP-28481 streamStatus probe should detect all abnormal computation termination

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          New Feature - A new feature of the product, which has yet to be developed. NXP-28043 Backport Stream Processor probe to the runningstatus

          • Major - Major loss of function.
          • Resolved

          Task - A task that needs to be done. NXDOC-1936 Add a Nuxeo Stream section about error handling

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. NXP-27529 Provide a recovery procedure for systematic failure in a stream processor

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. NXP-28094 Add Nuxeo Stream probe to health check by default

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Task - A task that needs to be done. NXP-27164 Add a Stream Processor probe to the runningstatus

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

            Activity

              People

              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - 0 minutes
                  0m
                  Remaining:
                  Remaining Estimate - 0 minutes
                  0m
                  Logged:
                  Time Spent - 6 hours
                  6h