Management API - Audit purge endpoint

One of the maintenance activities performed by cloud operations is Elasticsearch audit event trimming.  We remove the highest volume audit events for a few of the larger customers on a monthly basis to keep the volume/size at a reasonable level.  For example, an existing cloud customer keeps only the last 6 months worth of audit information, with focus on the largest volume events:

• documentCheckedIn
• documentModified
• documentCreated
• loginSuccess
• documentRemoved
• download

The audit events removed differ on a customer-by-customer basis.

The removal is achieved by manually running a shell script that does the following:

• Determines the total # of events eligible for removal based on date and event type(s)
  • Sample call =

    curl -XGET 'https://vpc-es-nfsecm-500-prodr-ahgsxo3ogzas64soc4szrq6jva.us-east-2.es.amazonaws.com/nuxeo-audit/_search?pretty&size=1' -H 'Content-Type: application/json' -d '{"query": {"bool": {"must": [{"terms": {"eventId": ["documentCreated","loginSuccess"]}},{"range": {"eventDate": {"lte": "2022-08-23"}}}]}}}'

• Makes a call to delete audit records with a given size threshold
  • Sample call =

    curl -XPOST "https://vpc-es-nfsecm-500-prodr-ahgsxo3ogzas64soc4szrq6jva.us-east-2.es.amazonaws.com/nuxeo-audit/_delete_by_query?pretty&size=50000000" -H 'Content-Type: application/json' -d '{"query": {"bool": {"must": [{"terms": {"eventId": ["documentUnlocked"]}},{"range": {"eventDate": {"lte": "2022-06-28"}}}]}}}'

To avoid having to run shell scripts manually on a customer-by-customer basis, cloud operations would like to be able to use the cloud console to streamline the process and make it more scalable/"operationally sound".  Furthermore, it would be nice to allow customers (cloud or on-premise) to perform this activity themselves.

To accomplish this, additions to the Elasticsearch management endpoint could be made:

POST /management/elasticsearch/audit-event-removal

Input Parameters

• Removal_Date (ex:  "2022-08-23")
• Audit_Event_Types (ex:  "documentCheckedIn, documentModified, loginSucess")
• Max_Count_to_Remove (ex:  50000000)

Output

• None

Notes

• Anything older than the provided removal date gets deleted
• Ability to provide 1-many audit event types
• Max count is intended to limit the # of audit events deleted in cases where there are large numbers and dealing with a defined period of time in a maintenance window.  Should be optional, if not provided all events are deleted

POST /management/elasticsearch/audit-event-count

Input Parameters

• Removal_Date (ex:  "2022-08-23")
• Audit_Event_Types (ex:  "documentCheckedIn, documentModified, loginSucess")

Output

• Count of events

Notes

• Ability to provide 1-many audit event types

POST /management/elasticsearch/audit-event-removal-status

Input

• None

Output

• JSON showing # of audit events deleted

Notes

• This is a way to determine the status of the audit event removal call after invoked as it will take time to run
• Sample call operations makes manually:

  curl -X GET "https://vpc-es-nfsecm-500-prodr-ahgsxo3ogzas64soc4szrq6jva.us-east-2.es.amazonaws.com/_tasks?detailed=true&actions=*/delete/byquery&pretty"

** Above intended to be a first draft of what would be required **