Uploaded image for project: 'Nuxeo Platform'
  1. Nuxeo Platform
  2. NXP-16396

Loading from Elasticsearch should allow to easily scale out

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 6.0-HF08, 7.2
    • Fix Version/s: 6.0-HF10, 7.3
    • Component/s: Elasticsearch

      Description

      The current way of loading DataModels from Elasticsearch will actually load the DataModels from the Core before overridding it by Elasticsearch data.

      Initial loading

      This is a side effect of the DocumentModel lazyLoading systems : when not initialized the DataModels are loaded from the core before any property can be written.

      The solution is to pre-initialized the DataModels :

      --- a/nuxeo-features/nuxeo-elasticsearch/nuxeo-elasticsearch-core/src/main/java/org/nuxeo/elasticsearch/io/JsonDocumentModelReader.java
+++ b/nuxeo-features/nuxeo-elasticsearch/nuxeo-elasticsearch-core/src/main/java/org/nuxeo/elasticsearch/io/JsonDocumentModelReader.java
@@ -25,9 +25,15 @@ import org.elasticsearch.search.lookup.SourceLookup;
 import org.nuxeo.common.utils.Path;
 import org.nuxeo.ecm.core.api.ClientException;
 import org.nuxeo.ecm.core.api.CoreSession;
+import org.nuxeo.ecm.core.api.DocumentException;
 import org.nuxeo.ecm.core.api.DocumentModel;
+import org.nuxeo.ecm.core.api.DocumentModelFactory;
 import org.nuxeo.ecm.core.api.IdRef;
 import org.nuxeo.ecm.core.api.impl.DocumentModelImpl;
+import org.nuxeo.ecm.core.schema.DocumentType;
+import org.nuxeo.ecm.core.schema.SchemaManager;
+import org.nuxeo.ecm.core.schema.types.Schema;
+import org.nuxeo.runtime.api.Framework;
 
 /**
  * Read a DocumentModel from an ES Json export.
@@ -71,6 +77,16 @@ public class JsonDocumentModelReader {
 
         DocumentModelImpl doc = new DocumentModelImpl(sid, getType(), id, new Path(path), new IdRef(id), new IdRef(
                 parentId), null, null, null, repository, false);
+        try {
+            // pre load 
+            DocumentType docType = Framework.getService(SchemaManager.class).getDocumentType(type);
+            for (Schema schema : docType.getSchemas()) {
+                doc.addDataModel(DocumentModelFactory.createDataModel(null, schema));
+            }
+        } catch (DocumentException e1) {
+            log.error("Unable to pre load DataModels", e1);
+        }        
+        
         for (String prop : source.keySet()) {
             String schema = prop.split(":")[0];

      additional loading

      When accessing DocumentModel via the REST API, the JSON Marshaling will read all attributes of the DocumentModel.

      To make this scale, we must be sure that :

      • All DocumentModel data can be loaded from ES
      • no locks are used

      After a quick check, some of the DocumentModel attributes are problematic :

      jg.writeStringField("versionLabel", doc.getVersionLabel());

      => access to the Core
      => AbstractSession.refreshDocument

      jg.writeBooleanField("isCheckedOut", doc.isCheckedOut());

      => access to the Core
      => AbstractSession.refreshDocument

      Lock lock = doc.getLockInfo();

      => Acces to a synchronized block in the LockManager
      ==> org.nuxeo.ecm.core.storage.sql.VCSLockManager.getLock

        Attachments

          Issue Links

          depends on

          Improvement - An improvement or enhancement to an existing feature or task. NXP-16429 Load versioning information from ES

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. NXP-16430 Move JSON Marshaling of the DocumentModel Lock to a Content Enricher

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. NXP-16711 Update the ES marshaller to use nuxeo-core-io system

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Open
          duplicates

          Improvement - An improvement or enhancement to an existing feature or task. NXP-15205 Improve performance of document fetching from ES

          • Major - Major loss of function.
          • Resolved
          is required by

          Bug - A problem which impairs or prevents the functions of the product. NXP-19039 fetchDocFromEs don't load complex properties

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. NXP-19443 Drive: optimize ScrollDescendants operation by avoiding DocumentModel loading

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. NXP-14374 Enable to load document from Elasticsearch source

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

            Activity

              People

              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: