Uploaded image for project: 'Nuxeo Drive '
  1. Nuxeo Drive
  2. NXDRIVE-2541

More efficient handling of local folder deletions

    Details

      Description

      Current Behavior

      When deleting a folder containing a lot of subfolder and files, the process will iterate over each and every subfolder and file.

      This is not efficient as Drive only needs to send the top-level folder deletion to the server. All other events will be ignored. But even if they will be ignored, there will be database access and several actions that will just consume resources for nothing.

      Here is a sample of logs (deleted the folder named "1000":

      ...
      2021-02-25 15:43:55 63227 123145564352512 DEBUG    nxdrive.engine.watcher.local_watcher Queueing watchdog: <FileDeletedEvent: src_path='/Users/tiger-222/Nuxeo Drive/1000/674.txt'>
      ...
      2021-02-25 15:43:55 63227 123145564352512 DEBUG    nxdrive.engine.watcher.local_watcher Queueing watchdog: <FileDeletedEvent: src_path='/Users/tiger-222/Nuxeo Drive/1000/900.txt'>
      2021-02-25 15:43:55 63227 123145564352512 DEBUG    nxdrive.engine.watcher.local_watcher Queueing watchdog: <DirDeletedEvent: src_path='/Users/tiger-222/Nuxeo Drive/1000'>
      
      2021-02-25 15:43:56 63227 123145462005760 INFO     nxdrive.engine.watcher.local_watcher Handling watchdog event [deleted] on '/Users/tiger-222/Nuxeo Drive/1000/674.txt'
      2021-02-25 15:43:56 63227 123145462005760 DEBUG    nxdrive.engine.watcher.local_watcher Watchdog event <FileDeletedEvent: src_path='/Users/tiger-222/Nuxeo Drive/1000/674.txt'> on known pair <DocPair[643] local_path=PosixPath('NXDRIVE-2479/1000/674.txt')...>
      2021-02-25 15:43:56 63227 123145462005760 DEBUG    nxdrive.engine.watcher.local_watcher Deleted event on inexistent file: PosixPath('/Users/tiger-222/Nuxeo Drive/1000/674.txt')
      ...
      2021-02-25 15:43:59 63227 123145462005760 INFO     nxdrive.engine.watcher.local_watcher Handling watchdog event [deleted] on '/Users/tiger-222/Nuxeo Drive/1000'
      2021-02-25 15:43:59 63227 123145462005760 DEBUG    nxdrive.engine.watcher.local_watcher Watchdog event <DirDeletedEvent: src_path='/Users/tiger-222/Nuxeo Drive/1000'> on known pair <DocPair[3] local_path=PosixPath('NXDRIVE-2479/1000')...>
      2021-02-25 15:43:59 63227 123145462005760 DEBUG    nxdrive.engine.watcher.local_watcher Deleting file: PosixPath('/Users/tiger-222/Nuxeo Drive/1000')
      

      The folder contained 1,000 files. There were then 999 useless events.

      New Behavior

      Making use of the local events queue (NXDRIVE-2540), such events could be ingested and one final FS event should be put into the LocalWatcher queue: the top-level folder.

        Attachments

          Issue Links

            Activity

              People

              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated:

                  PagerDuty

                  Error rendering 'com.pagerduty.jira-server-plugin:PagerDuty'. Please contact your Jira administrators.