Updating a document is a heavy operation:
- it creates a version
- it generates 3 audit log entries (documentCheckedIn, documentModified, documentCreated)
offset | watermark | flag | key | length | data |
---|---|---|---|---|---|
audit-00:+2783026 | 2018-10-17 17:26:02.897:0 | [DEFAULT] | 0 | 517 | {"entity-type":"logEntry","id":0,"category":"eventDocumentCategory","principalName":"Administrator","comment":"0.1","docLifeCycle":"project","docPath":"/default-domain/workspaces/Bench_Gatling/folder-828125/folder-1-0/folder-2-0/folder-3-0/folder-4-1/folder-5-0/folder-6-2/folder-7-2/folder-8-3/file-8-47","docType":"File","docUUID":"34688a8a-5dc8-402c-b189-31d478547ffa","eventId":"documentCheckedIn","repositoryId":"default","eventDate":"2018-10-17T17:26:02.897Z","logDate":"2018-10-17T17:26:02.988Z","extended":{}} |
audit-00:+2783027 | 2018-10-17 17:26:02.897:0 | [DEFAULT] | 0 | 515 | {"entity-type":"logEntry","id":0,"category":"eventDocumentCategory","principalName":"Administrator","comment":"0.1","docLifeCycle":"project","docPath":"/default-domain/workspaces/Bench_Gatling/folder-828125/folder-1-0/folder-2-0/folder-3-0/folder-4-1/folder-5-0/folder-6-2/folder-7-2/folder-8-3/file-8-47","docType":"File","docUUID":"c68839b5-a0ba-4644-a498-8a405fa1c2af","eventId":"documentCreated","repositoryId":"default","eventDate":"2018-10-17T17:26:02.897Z","logDate":"2018-10-17T17:26:02.989Z","extended":{}} |
audit-00:+2783028 | 2018-10-17 17:26:02.898:0 | [DEFAULT] | 0 | 515 | {"entity-type":"logEntry","id":0,"category":"eventDocumentCategory","principalName":"Administrator","comment":null,"docLifeCycle":"project","docPath":"/default-domain/workspaces/Bench_Gatling/folder-828125/folder-1-0/folder-2-0/folder-3-0/folder-4-1/folder-5-0/folder-6-2/folder-7-2/folder-8-3/file-8-47","docType":"File","docUUID":"34688a8a-5dc8-402c-b189-31d478547ffa","eventId":"documentModified","repositoryId":"default","eventDate":"2018-10-17T17:26:02.898Z","logDate":"2018-10-17T17:26:02.989Z","extended":{}} |
So when updating 1m docs, this means creating 1m versions, indexing 2m docs, log 3m lines.
We see that the main stress is not on the action but on async works generated, for 1m docs:
- the bulk command is completed within 14min: 1m at 1250 doc/s with 8 threads
- the async indexing takes 30min: 2m at 1600 docs/s with 12 threads
- the audit writer take 1h: 3m at 780 doc/s with a single thread
For mass update we want option to mitigate this and reduce the cost:
- an option to disable audit log
- an option to not create new version
Same problem and solution for setSystemProperties action.