-
Type: Improvement
-
Status: Resolved
-
Priority: Minor
-
Resolution: Fixed
-
Affects Version/s: 8.3
-
Fix Version/s: 9.1
-
Component/s: Core MongoDB
-
Epic Link:
-
Sprint:nxBD 8.4.6, nxBD 8.10.1, nxBD 9.1.4, nxBD 9.1.6, nxBD 9.1.7
-
Story Points:3
Ideally to get the higher import rate, we need to saturate the backend on disk IO. This is what we have when submitting only insert commands.
When we mix inserts and queries the backend CPU starts to limit the import rate. This is especially true on sharded backend where queries require additional communications between nodes.
For MongoDB this query CPU load on mongos and mongod prevents to take advantage of improved disk IO provided by a sharded cluster.
This can be improved by using a simple cache on states and other shortcut taken in this branch:
https://github.com/nuxeo/nuxeo/compare/test-NXBT-1103-import-tuning-1b?expand=1
Obviously using a cache in Nuxeo cluster mode will require an invalidation mechanism.
So If we can not merge this change as is, may be they can be activated conditionally.
–
To limit the number of backend queries during document creation or import, typically to improve performance on a mass import,
one can now add a flag to skip the document id duplication check:
doc.putContextData(CoreSession.SKIP_DESTINATION_CHECK_ON_CREATE, true);