Uploaded image for project: 'Nuxeo Platform'
  1. Nuxeo Platform
  2. NXP-19586

Drive: Implement Elasticsearch based batched remote scan

    XMLWordPrintable

    Details

    • Type: Task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 8.3
    • Component/s: Nuxeo Drive

      Description

      The VCS implementation of the batched remote scan, see NXP-19482 and DocumentBackedFolderItem#getScrollBatch relies on a "homemade" implementation of a scroll search and doesn't allow to sort the documents by path because it relies on a queryAndFecth core query.
      Yet on the Drive side we would rather have the documents sorted by path to easily build the remote tree information in the local storage (need the FileSystemItem path).

      That's why we now rely on the Elasticsearch scroll API exposed by the ElasticSearchService, see NXP-19631, in the case of a synchronization root and if Elasticsearch is enabled of course.
      See ESSyncRootFolderItem#getScrollBatch.
      This way we use the native ES scroll which allows to return the documents sorted by path.

      Note: we consider that when scrolling through the descendants of a synchronization root its sub-tree must be indexed. This is most likely the case when performing the initial remote scan or in the case of a newly registered folder.

        Attachments

          Issue Links

          depends on

          New Feature - A new feature of the product, which has yet to be developed. NXP-19631 Expose Elasticsearch Scroll API in ElasticSearchService

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Task - A task that needs to be done. NXP-19388 Setup a bench for the Nuxeo Drive optimized remote scan

          • Major - Major loss of function.
          • Resolved

          Task - A task that needs to be done. NXP-19482 Drive: Optimize remote scan execution by using a scroll API

          • Major - Major loss of function.
          • Resolved
          is required by

          Task - A task that needs to be done. NXP-19659 Limit number of threads calling DocumentBackedFolderItem#scrollDescendants

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. NXP-19443 Drive: optimize ScrollDescendants operation by avoiding DocumentModel loading

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. NXDRIVE-307 Remote scan: use new API to fetch the descendants of a folder

          • Major - Major loss of function.
          • Resolved

          User story - Created by Jira Software - do not edit or delete. Issue type for a user story. NXDRIVE-441 Test improved Remote Scan execution with volume

          • Major - Major loss of function.
          • Resolved

            Activity

              People

              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: