Uploaded image for project: 'Nuxeo Drive '
  1. Nuxeo Drive
  2. NXDRIVE-581

Optimize remote scan local processing: SQLite insertion duration is increasing along with the number of rows

    XMLWordPrintable

    Details

      Description

      When performing a remote scan of a large volume of documents, the duration of the SQLite DB insertions seem to increase along with the number of documents.
      This is visible when launching VolumeTestCase.test_remote_scan by setting the TEST_REMOTE_SCAN_VOLUME environment variable to <docCount>, activating the TRACE logs and looking at:

      TRACE    nxdrive.engine.watcher.remote_watcher Local processing of descendants of Nuxeo Drive Test Workspace (defaultSyncRootFolderItemFactory#default#1a7520cc-dd2f-45ff-9b52-03cbb201f888) took 606 ms
      

      Profiling RemoteWatcher._scan_remote_scroll shows that most of the time is spent in:

      _find_remote_child_match_or_create (70 %)
       --> insert_remote_state (47 %)
           --> commit (49 %)
      

      Attached profiling stats of a test with 1000 documents.

        Attachments

        1. pyvmmonitor_commit.png
          105 kB
          Antoine Taillefer
        2. pyvmmonitor_find_remote_child_match_or_create.png
          198 kB
          Antoine Taillefer
        3. pyvmmonitor_hotspots.png
          228 kB
          Antoine Taillefer
        4. remote_scan_profiling_1000_docs.pstat
          97 kB
          Antoine Taillefer

          Issue Links

            Activity

              People

              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0 minutes
                  0m
                  Logged:
                  Time Spent - 1 minute
                  1m