Uploaded image for project: 'Nuxeo Platform'
  1. Nuxeo Platform
  2. NXP-16396

Loading from Elasticsearch should allow to easily scale out



    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 6.0-HF08, 7.2
    • Fix Version/s: 6.0-HF10, 7.3
    • Component/s: Elasticsearch


      The current way of loading DataModels from Elasticsearch will actually load the DataModels from the Core before overridding it by Elasticsearch data.

      Initial loading

      This is a side effect of the DocumentModel lazyLoading systems : when not initialized the DataModels are loaded from the core before any property can be written.

      The solution is to pre-initialized the DataModels :

      --- a/nuxeo-features/nuxeo-elasticsearch/nuxeo-elasticsearch-core/src/main/java/org/nuxeo/elasticsearch/io/JsonDocumentModelReader.java
      +++ b/nuxeo-features/nuxeo-elasticsearch/nuxeo-elasticsearch-core/src/main/java/org/nuxeo/elasticsearch/io/JsonDocumentModelReader.java
      @@ -25,9 +25,15 @@ import org.elasticsearch.search.lookup.SourceLookup;
       import org.nuxeo.common.utils.Path;
       import org.nuxeo.ecm.core.api.ClientException;
       import org.nuxeo.ecm.core.api.CoreSession;
      +import org.nuxeo.ecm.core.api.DocumentException;
       import org.nuxeo.ecm.core.api.DocumentModel;
      +import org.nuxeo.ecm.core.api.DocumentModelFactory;
       import org.nuxeo.ecm.core.api.IdRef;
       import org.nuxeo.ecm.core.api.impl.DocumentModelImpl;
      +import org.nuxeo.ecm.core.schema.DocumentType;
      +import org.nuxeo.ecm.core.schema.SchemaManager;
      +import org.nuxeo.ecm.core.schema.types.Schema;
      +import org.nuxeo.runtime.api.Framework;
        * Read a DocumentModel from an ES Json export.
      @@ -71,6 +77,16 @@ public class JsonDocumentModelReader {
               DocumentModelImpl doc = new DocumentModelImpl(sid, getType(), id, new Path(path), new IdRef(id), new IdRef(
                       parentId), null, null, null, repository, false);
      +        try {
      +            // pre load 
      +            DocumentType docType = Framework.getService(SchemaManager.class).getDocumentType(type);
      +            for (Schema schema : docType.getSchemas()) {
      +                doc.addDataModel(DocumentModelFactory.createDataModel(null, schema));
      +            }
      +        } catch (DocumentException e1) {
      +            log.error("Unable to pre load DataModels", e1);
      +        }        
               for (String prop : source.keySet()) {
                   String schema = prop.split(":")[0];

      additional loading

      When accessing DocumentModel via the REST API, the JSON Marshaling will read all attributes of the DocumentModel.

      To make this scale, we must be sure that :

      • All DocumentModel data can be loaded from ES
      • no locks are used

      After a quick check, some of the DocumentModel attributes are problematic :

      jg.writeStringField("versionLabel", doc.getVersionLabel());

      => access to the Core
      => AbstractSession.refreshDocument

      jg.writeBooleanField("isCheckedOut", doc.isCheckedOut());

      => access to the Core
      => AbstractSession.refreshDocument

      Lock lock = doc.getLockInfo();

      => Acces to a synchronized block in the LockManager
      ==> org.nuxeo.ecm.core.storage.sql.VCSLockManager.getLock


          Issue Links



              • Votes:
                0 Vote for this issue
                4 Start watching this issue


                • Created: