Uploaded image for project: 'Nuxeo Platform'
  1. Nuxeo Platform
  2. NXP-20595

Reduce the number of DBS queries during document creation

    XMLWordPrintable

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 8.3
    • Fix Version/s: 9.1
    • Component/s: Core MongoDB

      Description

      Ideally to get the higher import rate, we need to saturate the backend on disk IO. This is what we have when submitting only insert commands.

      When we mix inserts and queries the backend CPU starts to limit the import rate. This is especially true on sharded backend where queries require additional communications between nodes.
      For MongoDB this query CPU load on mongos and mongod prevents to take advantage of improved disk IO provided by a sharded cluster.

      This can be improved by using a simple cache on states and other shortcut taken in this branch:
      https://github.com/nuxeo/nuxeo/compare/test-NXBT-1103-import-tuning-1b?expand=1

      Obviously using a cache in Nuxeo cluster mode will require an invalidation mechanism.

      So If we can not merge this change as is, may be they can be activated conditionally.

      To limit the number of backend queries during document creation or import, typically to improve performance on a mass import,
      one can now add a flag to skip the document id duplication check:

      doc.putContextData(CoreSession.SKIP_DESTINATION_CHECK_ON_CREATE, true);
      

        Attachments

          Issue Links

            Activity

              People

              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: