Uploaded image for project: 'Nuxeo Platform'
  1. Nuxeo Platform
  2. NXP-30565

Avoid race condition during Elasticsearch index initialization

    XMLWordPrintable

    Details

    • Release Notes Summary:
      Concurrent Elasticsearch index initialization is managed.
    • Team:
      PLATFORM
    • Sprint:
      nxplatform #69
    • Story Points:
      3

      Description

      We have seen the errors below which could indicate a concurrent race condition on Nuxeo nodes startup:

      org.nuxeo.ecm.core.api.NuxeoException: org.elasticsearch.client.ResponseException: method [PUT], host [https://*****], URI [/nuxeo-audit?timeout=60s], status line [HTTP/1.1 400 Bad Request]
      {"error":{"root_cause":[{"type":"resource_already_exists_exception","reason":"index [nuxeo-audit/zARkteD6Ty6bR2m_qQk-9A] already exists","index_uuid":"zARkteD6Ty6bR2m_qQk-9A","index":"nuxeo-audit"}],"type":"resource_already_exists_exception","reason":"index [nuxeo-audit/zARkteD6Ty6bR2m_qQk-9A] already exists","index_uuid":"zARkteD6Ty6bR2m_qQk-9A","index":"nuxeo-audit"},"status":400}
      	at org.nuxeo.elasticsearch.client.ESRestClient.performRequest(ESRestClient.java:205) ~[nuxeo-elasticsearch-core-10.10-HF39.jar:?]
      	at org.nuxeo.elasticsearch.client.ESRestClient.createIndex(ESRestClient.java:185) ~[nuxeo-elasticsearch-core-10.10-HF39.jar:?]
      	at org.nuxeo.elasticsearch.core.ElasticSearchAdminImpl.initIndex(ElasticSearchAdminImpl.java:471) ~[nuxeo-elasticsearch-core-10.10-HF39.jar:?]
      	at org.nuxeo.elasticsearch.core.ElasticSearchAdminImpl.initIndex(ElasticSearchAdminImpl.java:368) ~[nuxeo-elasticsearch-core-10.10-HF39.jar:?]
      	at org.nuxeo.elasticsearch.core.ElasticSearchAdminImpl.initIndex(ElasticSearchAdminImpl.java:353) ~[nuxeo-elasticsearch-core-10.10-HF39.jar:?]
      	at org.nuxeo.elasticsearch.core.ElasticSearchAdminImpl.initIndexes(ElasticSearchAdminImpl.java:317) ~[nuxeo-elasticsearch-core-10.10-HF39.jar:?]
      	at org.nuxeo.elasticsearch.core.ElasticSearchAdminImpl.initializeIndexes(ElasticSearchAdminImpl.java:200) ~[nuxeo-elasticsearch-core-10.10-HF39.jar:?]
      	at org.nuxeo.elasticsearch.core.ElasticSearchAdminImpl.<init>(ElasticSearchAdminImpl.java:106) ~[nuxeo-elasticsearch-core-10.10-HF39.jar:?]
      	at org.nuxeo.elasticsearch.ElasticSearchComponent.start(ElasticSearchComponent.java:181) ~[nuxeo-elasticsearch-core-10.10-HF39.jar:?]
      	at org.nuxeo.runtime.model.impl.RegistrationInfoImpl.start(RegistrationInfoImpl.java:381) [nuxeo-runtime-10.10-HF10.jar:?]
      	at org.nuxeo.runtime.model.impl.ComponentManagerImpl.startComponent(ComponentManagerImpl.java:709) [nuxeo-runtime-10.10-HF10.jar:?]
      	at org.nuxeo.runtime.model.impl.ComponentManagerImpl.startComponents(ComponentManagerImpl.java:691) [nuxeo-runtime-10.10-HF10.jar:?]
      	at org.nuxeo.runtime.model.impl.ComponentManagerImpl.start(ComponentManagerImpl.java:792) [nuxeo-runtime-10.10-HF10.jar:?]
      	at org.nuxeo.runtime.osgi.OSGiRuntimeService.startComponents(OSGiRuntimeService.java:447) [nuxeo-runtime-10.10-HF10.jar:?]
      	at org.nuxeo.runtime.osgi.OSGiRuntimeService.frameworkEvent(OSGiRuntimeService.java:462) [nuxeo-runtime-10.10-HF10.jar:?]
      	at org.nuxeo.osgi.OSGiAdapter.fireFrameworkEvent(OSGiAdapter.java:223) [nuxeo-runtime-osgi-10.10.jar:?]
      	at org.nuxeo.osgi.application.loader.FrameworkLoader.doStart(FrameworkLoader.java:225) [nuxeo-runtime-osgi-10.10.jar:?]
      	at org.nuxeo.osgi.application.loader.FrameworkLoader.start(FrameworkLoader.java:124) [nuxeo-runtime-osgi-10.10.jar:?]
      	at org.nuxeo.runtime.deployment.NuxeoStarter.start(NuxeoStarter.java:124) [nuxeo-runtime-deploy-10.10.jar:?]
      	at org.nuxeo.runtime.deployment.NuxeoStarter.contextInitialized(NuxeoStarter.java:93) [nuxeo-runtime-deploy-10.10.jar:?]
      	at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4716) [catalina-9.0.43.jar:9.0.43]
      	at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5177) [catalina-9.0.43.jar:9.0.43]
      	at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) [catalina-9.0.43.jar:9.0.43]
      	at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:717) [catalina-9.0.43.jar:9.0.43]
      	at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:690) [catalina-9.0.43.jar:9.0.43]
      	at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:706) [catalina-9.0.43.jar:9.0.43]
      	at org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:623) [catalina-9.0.43.jar:9.0.43]
      	at org.apache.catalina.startup.HostConfig$DeployDescriptor.run(HostConfig.java:1774) [catalina-9.0.43.jar:9.0.43]
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_282]
      	at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_282]
      	at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75) [tomcat-util-9.0.43.jar:9.0.43]
      	at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) [?:1.8.0_282]
      	at org.apache.catalina.startup.HostConfig.deployDescriptors(HostConfig.java:526) [catalina-9.0.43.jar:9.0.43]
      	at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:424) [catalina-9.0.43.jar:9.0.43]
      	at org.apache.catalina.startup.HostConfig.start(HostConfig.java:1520) [catalina-9.0.43.jar:9.0.43]
      	at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:309) [catalina-9.0.43.jar:9.0.43]
      	at org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:123) [catalina-9.0.43.jar:9.0.43]
      	at org.apache.catalina.util.LifecycleBase.setStateInternal(LifecycleBase.java:423) [catalina-9.0.43.jar:9.0.43]
      	at org.apache.catalina.util.LifecycleBase.setState(LifecycleBase.java:366) [catalina-9.0.43.jar:9.0.43]
      	at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:936) [catalina-9.0.43.jar:9.0.43]
      	at org.apache.catalina.core.StandardHost.startInternal(StandardHost.java:843) [catalina-9.0.43.jar:9.0.43]
      	at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) [catalina-9.0.43.jar:9.0.43]
      	at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1384) [catalina-9.0.43.jar:9.0.43]
      	at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1374) [catalina-9.0.43.jar:9.0.43]
      	at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_282]
      	at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75) [tomcat-util-9.0.43.jar:9.0.43]
      	at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:134) [?:1.8.0_282]
      	at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:909) [catalina-9.0.43.jar:9.0.43]
      	at org.apache.catalina.core.StandardEngine.startInternal(StandardEngine.java:262) [catalina-9.0.43.jar:9.0.43]
      	at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) [catalina-9.0.43.jar:9.0.43]
      	at org.apache.catalina.core.StandardService.startInternal(StandardService.java:434) [catalina-9.0.43.jar:9.0.43]
      	at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) [catalina-9.0.43.jar:9.0.43]
      	at org.apache.catalina.core.StandardServer.startInternal(StandardServer.java:930) [catalina-9.0.43.jar:9.0.43]
      	at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) [catalina-9.0.43.jar:9.0.43]
      	at org.apache.catalina.startup.Catalina.start(Catalina.java:772) [catalina-9.0.43.jar:9.0.43]
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_282]
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_282]
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_282]
      	at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_282]
      	at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:342) [bootstrap-9.0.43.jar:9.0.43]
      	at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:473) [bootstrap-9.0.43.jar:9.0.43]
      

      Goal is to add a cluster wide lock to do the index initialization.

        Attachments

          Issue Links

            Activity

              People

              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: