Uploaded image for project: 'Nuxeo Drive '
  1. Nuxeo Drive
  2. NXDRIVE-581

Optimize remote scan local processing: SQLite insertion duration is increasing along with the number of rows

    XMLWordPrintable

    Details

      Description

      When performing a remote scan of a large volume of documents, the duration of the SQLite DB insertions seem to increase along with the number of documents.
      This is visible when launching VolumeTestCase.test_remote_scan by setting the TEST_REMOTE_SCAN_VOLUME environment variable to <docCount>, activating the TRACE logs and looking at:

      TRACE    nxdrive.engine.watcher.remote_watcher Local processing of descendants of Nuxeo Drive Test Workspace (defaultSyncRootFolderItemFactory#default#1a7520cc-dd2f-45ff-9b52-03cbb201f888) took 606 ms

      Profiling RemoteWatcher._scan_remote_scroll shows that most of the time is spent in:

      _find_remote_child_match_or_create (70 %)
 --> insert_remote_state (47 %)
     --> commit (49 %)

      Attached profiling stats of a test with 1000 documents.

        Attachments

        1. pyvmmonitor_commit.png
          105 kB
          Antoine Taillefer
        2. pyvmmonitor_find_remote_child_match_or_create.png
          198 kB
          Antoine Taillefer
        3. pyvmmonitor_hotspots.png
          228 kB
          Antoine Taillefer
        4. remote_scan_profiling_1000_docs.pstat
          97 kB
          Antoine Taillefer

          Issue Links

          depends on

          New Feature - A new feature of the product, which has yet to be developed. NXDRIVE-1921 Add the capability to monitor tests

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Open

          User story - Created by Jira Software - do not edit or delete. Issue type for a user story. NXDRIVE-441 Test improved Remote Scan execution with volume

          • Major - Major loss of function.
          • Resolved
          is related to

          Bug - A problem which impairs or prevents the functions of the product. NXDRIVE-1888 Improve database queries to anihile any deadlock

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

            Activity

              People

              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0 minutes
                  0m
                  Logged:
                  Time Spent - 1 minute
                  1m