Uploaded image for project: 'Nuxeo Platform'
  1. Nuxeo Platform
  2. NXP-29078

Stream AuditLogWriter processing cannot be parallelized

    XMLWordPrintable

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 10.10
    • Fix Version/s: 10.10-HF50, 11.x, 2021.7
    • Component/s: Audit, Streams
    • Release Notes Summary:
      Enable concurrency on AuditLogWriter
    • Backlog priority:
      800
    • Team:
      PLATFORM
    • Sprint:
      nxplatform #38, nxplatform #39, nxplatform #40
    • Story Points:
      1

      Description

      Today the audit event listener (StreamAuditEventListener) in charge of writing the log entries to the audit stream is using the log entry id as a key to route the record to a partition.

      But the log entry id at this stage is always 0 resulting in routing all the records to a single partition. This prevents using more than a single consumer in the cluster.

      This is a problem only when the audit is specifically configured to use more than a single partition (default).

      There are cases where we may want to use more than a single partition:

      • consumers don't care about the ordering, this is the case for the audit log writer
      • support higher throughput on audit log writer
         

      Note that using more than one partition is not something that we want by default, this configuration provides a good default:

      • preserve a total ordering of log entry events (even with multiple time desynchronized nodes)
      • a good constant throughput without using more than 1 thread on the cluster
      • prevent stressing the audit backend (elasticsearch) too much

      Note that this is specific to 10.10 (LTS 2021 can use more than one partition) it also affects LTS 2021

        Attachments

          Activity

            People

            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: