Uploaded image for project: 'Nuxeo Platform'
  1. Nuxeo Platform
  2. NXP-21284

On a 100M documents repository, server can't start because of nuxeo-dam

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 9.2
    • Component/s: Core MongoDB, DAM
    • Tags:
    • Backlog priority:
      1,000
    • Upgrade notes:
      Hide

      Since 9.2, Picture facet is not dynamic anymore due to performance issue on big volume.
      If you want to make it dynamic again, you can override the facet definition and set perDocumentQuery to true, like this:
      <facet name="Picture" perDocumentQuery="true">
      <schema name="file"/>
      <schema name="picture"/>
      <schema name="image_metadata"/>
      </facet>

      You'll need to require this component in your contribution "org.nuxeo.ecm.platform.picture.coreTypes".

      Show
      Since 9.2, Picture facet is not dynamic anymore due to performance issue on big volume. If you want to make it dynamic again, you can override the facet definition and set perDocumentQuery to true, like this: <facet name="Picture" perDocumentQuery="true"> <schema name="file"/> <schema name="picture"/> <schema name="image_metadata"/> </facet> You'll need to require this component in your contribution "org.nuxeo.ecm.platform.picture.coreTypes".
    • Sprint:
      nxcore 9.2.2, nxcore 9.2.5, nxcore 9.2.3, nxcore 9.2.4, nxcore 9.2.6
    • Story Points:
      1

      Description

      Load test with 100 Million documents and MongoDB as backend (and standalone Elasticsearch).

      The 100Millions documents are File, created with the bulk importer (random creation). After the repository is filled, when I install nuxeo-dam, the startup fails after 4 mn with a MongoDB timeout error (see log)

      2016-12-05 19:19:47,434 INFO  [localhost-startStop-1] [org.mongodb.driver.connection] Opened connection [connectionId{localValue:2, serverValue:35402}] to 10.0.0.79:27017
2016-12-05 19:23:47,893 INFO  [localhost-startStop-1] [org.mongodb.driver.connection] Closed connection [connectionId{localValue:2, serverValue:35402}] to 10.0.0.79:27017 because there was a socket exception raised by this connection.
2016-12-05 19:23:47,899 INFO  [localhost-startStop-1] [org.mongodb.driver.connection] Opened connection [connectionId{localValue:3, serverValue:35435}] to 10.0.0.79:27017
2016-12-05 19:23:47,914 ERROR [localhost-startStop-1] [org.nuxeo.runtime.model.impl.RegistrationInfoImpl] Component service:org.nuxeo.ecm.core.repository.RepositoryServiceComponent notification of application started failed: Timeout while receiving message
com.mongodb.MongoSocketReadTimeoutException: Timeout while receiving message

      The guilty is PictureMigrationHandler
      => After disabling it (nuxeo.picture.migration.enabled set to false in an xml contribution), problem is solved.

      We suspect this has to do with the query against MongoDB:

      SELECT ecm:uuid FROM Document WHERE ecm:mixinType = 'Picture' AND ecm:isProxy = 0 AND views/*/title = 'Original' AND content/data IS NULL

        Attachments

          Issue Links

          depends on

          New Feature - A new feature of the product, which has yet to be developed. NXP-19412 Migration Service

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. NXP-22603 Make perDocumentQuery attribute of facet descriptor overridable

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved
          is related to

          Improvement - An improvement or enhancement to an existing feature or task. NXP-19812 Make picture migration at startup optional

          • Major - Major loss of function.
          • Resolved

            Activity

              People

              • Votes:
                0 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0 minutes
                  0m
                  Logged:
                  Time Spent - 6 hours
                  6h