[NXP-29713] Make TransientStorageComponent.doGC() + Redis compatible with Hot-Reload - Nuxeo Issue Tracker

XML

Word

Printable

Details

Type: Bug
Status: Resolved
Priority: Minor
Resolution: Won't Fix
Affects Version/s: 9.10
Fix Version/s: None
Component/s: TransientStore
Environment:
Redis-enabled environment

Tags:
- SupCom

Description

Steps to reproduce:

enable REDIS,
start Nuxeo,
HotReload a project,
wait for up to 15 minutes

=> You will get exceptions such as the following gotten in Nuxeo 9.10

2020-09-29 10:47:56,007 INFO  [ReloadComponent] Before updating Nuxeo server
======================================================================
= Component Loading Status: Pending: 0 / Missing: 0 / Unstarted: 0 / Total: 536
======================================================================
2020-09-29 10:47:56,009 INFO  [ReloadComponent] Before flush caches
2020-09-29 10:47:56,017 INFO  [ReloadComponent] Before flush the JAAS cache
2020-09-29 10:47:56,017 INFO  [ReloadComponent] After flush the JAAS cache
2020-09-29 10:47:56,017 INFO  [ReloadComponent] After flush caches
2020-09-29 10:47:56,017 INFO  [ReloadComponent] Component reload strategy=standby
2020-09-29 10:47:56,017 INFO  [ReloadComponent] Before stop/standby component manager
2020-09-29 10:47:56,330 INFO  [ReloadComponent] After stop/standby component manager
2020-09-29 10:47:56,330 INFO  [ReloadComponent] Before undeploy bundles
2020-09-29 10:47:56,330 INFO  [ReloadComponent] Before undeploy bundle with name 'studio.extensions.pabgrall-sandbox-92-nuxe'.
2020-09-29 10:47:56,330 INFO  [ReloadComponent] After undeploy bundle with name 'studio.extensions.pabgrall-sandbox-92-nuxe'.
2020-09-29 10:47:56,330 INFO  [ReloadComponent] After undeploy bundles
2020-09-29 10:47:56,330 INFO  [ReloadComponent] Before delete-copy
2020-09-29 10:47:56,330 INFO  [ReloadComponent] After delete-copy
2020-09-29 10:47:56,330 INFO  [ReloadComponent] Before deploy bundles
2020-09-29 10:47:56,330 INFO  [ReloadComponent] Before deploy bundle for file at 'C:\CASES\SUP31448\t\nuxeo-server-9.10-tomcat\nxserver\bundles\pabgrall-sandbox-92-nuxe-master-SNAPSHOT.jar'
2020-09-29 10:47:56,347 INFO  [ReloadComponent] Deploy done for bundle with name 'studio.extensions.pabgrall-sandbox-92-nuxe'
2020-09-29 10:47:56,363 INFO  [ReloadComponent] After deploy bundles
2020-09-29 10:47:56,363 INFO  [ReloadComponent] Before start/resume component manager
2020-09-29 10:47:57,107 WARN  [ElasticSearchEmbeddedNode] Elasticsearch embedded configuration is ONLY for testing purpose. You need to create a dedicated Elasticsearch cluster for production.
2020-09-29 10:48:02,772 INFO  [ReloadComponent] After start/resume component manager
2020-09-29 10:48:02,772 INFO  [ReloadComponent] Start running deployment preprocessor
2020-09-29 10:48:04,877 INFO  [ReloadComponent] Deployment preprocessing done
2020-09-29 10:48:04,877 INFO  [ReloadComponent] Before reload runtime properties
2020-09-29 10:48:04,880 INFO  [ReloadComponent] After reload runtime properties
2020-09-29 10:48:04,880 INFO  [ReloadComponent] After updating Nuxeo server
======================================================================
= Component Loading Status: Pending: 0 / Missing: 0 / Unstarted: 0 / Total: 536
======================================================================
2020-09-29 10:48:04,881 INFO  [ReloadComponent] Hot reload was done in 8873 ms, detailed steps:
- flush: 8 ms
- stop/standby: 315 ms
- undeploy-bundles: 4 ms
- delete-copy: 5 ms
- reload-resources: 0 ms
- deploy-bundles: 23 ms
- start/resume: 6404 ms
- deployment-preprocessor: 2105 ms
- reload-properties: 2 ms
2020-09-29 10:49:50,690 INFO  [ReloadComponent] Before updating Nuxeo server
======================================================================
= Component Loading Status: Pending: 0 / Missing: 0 / Unstarted: 0 / Total: 536
======================================================================
2020-09-29 10:49:50,690 INFO  [ReloadComponent] Before flush caches
2020-09-29 10:49:50,695 INFO  [ReloadComponent] Before flush the JAAS cache
2020-09-29 10:49:50,695 INFO  [ReloadComponent] After flush the JAAS cache
2020-09-29 10:49:50,695 INFO  [ReloadComponent] After flush caches
2020-09-29 10:49:50,695 INFO  [ReloadComponent] Component reload strategy=standby
2020-09-29 10:49:50,695 INFO  [ReloadComponent] Before stop/standby component manager
2020-09-29 10:49:50,971 INFO  [ReloadComponent] After stop/standby component manager
2020-09-29 10:49:50,971 INFO  [ReloadComponent] Before undeploy bundles
2020-09-29 10:49:50,971 INFO  [ReloadComponent] Before undeploy bundle with name 'studio.extensions.pabgrall-sandbox-92-nuxe'.
2020-09-29 10:49:50,972 INFO  [ReloadComponent] After undeploy bundle with name 'studio.extensions.pabgrall-sandbox-92-nuxe'.
2020-09-29 10:49:50,972 INFO  [ReloadComponent] After undeploy bundles
2020-09-29 10:49:50,972 INFO  [ReloadComponent] Before delete-copy
2020-09-29 10:49:50,976 INFO  [ReloadComponent] After delete-copy
2020-09-29 10:49:50,977 INFO  [ReloadComponent] Before deploy bundles
2020-09-29 10:49:50,977 INFO  [ReloadComponent] Before deploy bundle for file at 'C:\CASES\SUP31448\t\nuxeo-server-9.10-tomcat\nxserver\bundles\pabgrall-sandbox-92-nuxe-master-SNAPSHOT.jar'
2020-09-29 10:49:50,982 INFO  [ReloadComponent] Deploy done for bundle with name 'studio.extensions.pabgrall-sandbox-92-nuxe'
2020-09-29 10:49:50,995 INFO  [ReloadComponent] After deploy bundles
2020-09-29 10:49:50,995 INFO  [ReloadComponent] Before start/resume component manager
2020-09-29 10:49:51,595 WARN  [ElasticSearchEmbeddedNode] Elasticsearch embedded configuration is ONLY for testing purpose. You need to create a dedicated Elasticsearch cluster for production.
2020-09-29 10:49:56,517 INFO  [ReloadComponent] After start/resume component manager
2020-09-29 10:49:56,519 INFO  [ReloadComponent] Start running deployment preprocessor
2020-09-29 10:49:58,380 INFO  [ReloadComponent] Deployment preprocessing done
2020-09-29 10:49:58,380 INFO  [ReloadComponent] Before reload runtime properties
2020-09-29 10:49:58,381 INFO  [ReloadComponent] After reload runtime properties
2020-09-29 10:49:58,382 INFO  [ReloadComponent] After updating Nuxeo server
======================================================================
= Component Loading Status: Pending: 0 / Missing: 0 / Unstarted: 0 / Total: 536
======================================================================
2020-09-29 10:49:58,382 INFO  [ReloadComponent] Hot reload was done in 7692 ms, detailed steps:
- flush: 5 ms
- stop/standby: 275 ms
- undeploy-bundles: 0 ms
- delete-copy: 4 ms
- reload-resources: 0 ms
- deploy-bundles: 18 ms
- start/resume: 5524 ms
- deployment-preprocessor: 1860 ms
- reload-properties: 1 ms
2020-09-29 11:00:00,035 WARN  [SimpleDelay] pausing for 2000 ms
2020-09-29 11:00:02,049 WARN  [SimpleDelay] pausing for 4000 ms
2020-09-29 11:00:06,052 WARN  [SimpleDelay] pausing for 8000 ms
2020-09-29 11:00:14,068 WARN  [SimpleDelay] pausing for 16000 ms
2020-09-29 11:00:30,077 WARN  [SimpleDelay] pausing for 32000 ms
2020-09-29 11:01:02,083 WARN  [SimpleDelay] pausing for 64000 ms
2020-09-29 11:02:06,099 WARN  [SimpleDelay] pausing for 128000 ms
2020-09-29 11:04:14,106 WARN  [SimpleDelay] pausing for 256000 ms
2020-09-29 11:08:30,109 ERROR [AbstractWork] Exception during work: TransiantStorageGCWork(, Progress(?%, ?/0), null)
redis.clients.jedis.exceptions.JedisConnectionException: Cannot reconnect to jedis ..
        at org.nuxeo.ecm.core.redis.RedisFailoverExecutor.executeWithRetryPolicy(RedisFailoverExecutor.java:70)
        at org.nuxeo.ecm.core.redis.RedisFailoverExecutor.execute(RedisFailoverExecutor.java:43)
        at org.nuxeo.ecm.core.redis.contribs.RedisTransientStore.setStorageSize(RedisTransientStore.java:424)
        at org.nuxeo.ecm.core.transientstore.AbstractTransientStore.doGC(AbstractTransientStore.java:255)
        at java.util.HashMap$Values.forEach(HashMap.java:981)
        at org.nuxeo.ecm.core.transientstore.TransientStorageComponent.doGC(TransientStorageComponent.java:74)
        at org.nuxeo.ecm.core.transientstore.TransiantStorageGCWork.work(TransiantStorageGCWork.java:41)
        at org.nuxeo.ecm.core.work.AbstractWork.runWorkWithTransaction(AbstractWork.java:491)
        at org.nuxeo.ecm.core.work.AbstractWork.run(AbstractWork.java:381)
        at org.nuxeo.ecm.core.work.WorkHolder.run(WorkHolder.java:57)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: org.nuxeo.ecm.core.redis.retry.Retry$FailException: Cannot execute block, retry policy failed, check suppressed exception for more infos
        at org.nuxeo.ecm.core.redis.retry.Retry.retry(Retry.java:60)
        at org.nuxeo.ecm.core.redis.RedisFailoverExecutor.executeWithRetryPolicy(RedisFailoverExecutor.java:57)
        ... 12 more
        Suppressed: redis.clients.jedis.exceptions.JedisConnectionException: Could not get a resource from the pool
                at redis.clients.util.Pool.getResource(Pool.java:53)
                at redis.clients.jedis.JedisPool.getResource(JedisPool.java:226)
                at redis.clients.jedis.JedisPool.getResource(JedisPool.java:16)
                at org.nuxeo.ecm.core.redis.RedisPoolExecutor.execute(RedisPoolExecutor.java:56)
                at org.nuxeo.ecm.core.redis.RedisFailoverExecutor$1.retry(RedisFailoverExecutor.java:62)
                at org.nuxeo.ecm.core.redis.retry.Retry.retry(Retry.java:64)
                ... 13 more
        Caused by: java.lang.IllegalStateException: Pool not open
                at org.apache.commons.pool2.impl.BaseGenericObjectPool.assertOpen(BaseGenericObjectPool.java:672)
                at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:412)
                at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:363)
                at redis.clients.util.Pool.getResource(Pool.java:49)
                ... 18 more
        Suppressed: redis.clients.jedis.exceptions.JedisConnectionException: Could not get a resource from the pool
                at redis.clients.util.Pool.getResource(Pool.java:53)
                at redis.clients.jedis.JedisPool.getResource(JedisPool.java:226)
                at redis.clients.jedis.JedisPool.getResource(JedisPool.java:16)
                at org.nuxeo.ecm.core.redis.RedisPoolExecutor.execute(RedisPoolExecutor.java:56)
                at org.nuxeo.ecm.core.redis.RedisFailoverExecutor$1.retry(RedisFailoverExecutor.java:62)
                at org.nuxeo.ecm.core.redis.retry.Retry.retry(Retry.java:64)
                ... 13 more
        Caused by: java.lang.IllegalStateException: Pool not open
                at org.apache.commons.pool2.impl.BaseGenericObjectPool.assertOpen(BaseGenericObjectPool.java:672)
                at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:412)
                at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:363)
                at redis.clients.util.Pool.getResource(Pool.java:49)
                ... 18 more
        Suppressed: redis.clients.jedis.exceptions.JedisConnectionException: Could not get a resource from the pool
                at redis.clients.util.Pool.getResource(Pool.java:53)
                at redis.clients.jedis.JedisPool.getResource(JedisPool.java:226)
                at redis.clients.jedis.JedisPool.getResource(JedisPool.java:16)
                at org.nuxeo.ecm.core.redis.RedisPoolExecutor.execute(RedisPoolExecutor.java:56)
                at org.nuxeo.ecm.core.redis.RedisFailoverExecutor$1.retry(RedisFailoverExecutor.java:62)
                at org.nuxeo.ecm.core.redis.retry.Retry.retry(Retry.java:64)
                ... 13 more
        Caused by: java.lang.IllegalStateException: Pool not open
                at org.apache.commons.pool2.impl.BaseGenericObjectPool.assertOpen(BaseGenericObjectPool.java:672)
                at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:412)
                at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:363)
                at redis.clients.util.Pool.getResource(Pool.java:49)
                ... 18 more
        Suppressed: redis.clients.jedis.exceptions.JedisConnectionException: Could not get a resource from the pool
                at redis.clients.util.Pool.getResource(Pool.java:53)
                at redis.clients.jedis.JedisPool.getResource(JedisPool.java:226)
                at redis.clients.jedis.JedisPool.getResource(JedisPool.java:16)
                at org.nuxeo.ecm.core.redis.RedisPoolExecutor.execute(RedisPoolExecutor.java:56)
                at org.nuxeo.ecm.core.redis.RedisFailoverExecutor$1.retry(RedisFailoverExecutor.java:62)
                at org.nuxeo.ecm.core.redis.retry.Retry.retry(Retry.java:64)
                ... 13 more
        Caused by: java.lang.IllegalStateException: Pool not open
                at org.apache.commons.pool2.impl.BaseGenericObjectPool.assertOpen(BaseGenericObjectPool.java:672)
                at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:412)
                at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:363)
                at redis.clients.util.Pool.getResource(Pool.java:49)
                ... 18 more
        Suppressed: redis.clients.jedis.exceptions.JedisConnectionException: Could not get a resource from the pool
                at redis.clients.util.Pool.getResource(Pool.java:53)
                at redis.clients.jedis.JedisPool.getResource(JedisPool.java:226)
                at redis.clients.jedis.JedisPool.getResource(JedisPool.java:16)
                at org.nuxeo.ecm.core.redis.RedisPoolExecutor.execute(RedisPoolExecutor.java:56)
                at org.nuxeo.ecm.core.redis.RedisFailoverExecutor$1.retry(RedisFailoverExecutor.java:62)
                at org.nuxeo.ecm.core.redis.retry.Retry.retry(Retry.java:64)
                ... 13 more
        Caused by: java.lang.IllegalStateException: Pool not open
                at org.apache.commons.pool2.impl.BaseGenericObjectPool.assertOpen(BaseGenericObjectPool.java:672)
                at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:412)
                at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:363)
                at redis.clients.util.Pool.getResource(Pool.java:49)
                ... 18 more
        Suppressed: redis.clients.jedis.exceptions.JedisConnectionException: Could not get a resource from the pool
                at redis.clients.util.Pool.getResource(Pool.java:53)
                at redis.clients.jedis.JedisPool.getResource(JedisPool.java:226)
                at redis.clients.jedis.JedisPool.getResource(JedisPool.java:16)
                at org.nuxeo.ecm.core.redis.RedisPoolExecutor.execute(RedisPoolExecutor.java:56)
                at org.nuxeo.ecm.core.redis.RedisFailoverExecutor$1.retry(RedisFailoverExecutor.java:62)
                at org.nuxeo.ecm.core.redis.retry.Retry.retry(Retry.java:64)
                ... 13 more
        Caused by: java.lang.IllegalStateException: Pool not open
                at org.apache.commons.pool2.impl.BaseGenericObjectPool.assertOpen(BaseGenericObjectPool.java:672)
                at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:412)
                at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:363)
                at redis.clients.util.Pool.getResource(Pool.java:49)
                ... 18 more
        Suppressed: redis.clients.jedis.exceptions.JedisConnectionException: Could not get a resource from the pool
                at redis.clients.util.Pool.getResource(Pool.java:53)
                at redis.clients.jedis.JedisPool.getResource(JedisPool.java:226)
                at redis.clients.jedis.JedisPool.getResource(JedisPool.java:16)
                at org.nuxeo.ecm.core.redis.RedisPoolExecutor.execute(RedisPoolExecutor.java:56)
                at org.nuxeo.ecm.core.redis.RedisFailoverExecutor$1.retry(RedisFailoverExecutor.java:62)
                at org.nuxeo.ecm.core.redis.retry.Retry.retry(Retry.java:64)
                ... 13 more
        Caused by: java.lang.IllegalStateException: Pool not open
                at org.apache.commons.pool2.impl.BaseGenericObjectPool.assertOpen(BaseGenericObjectPool.java:672)
                at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:412)
                at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:363)
                at redis.clients.util.Pool.getResource(Pool.java:49)
                ... 18 more
        Suppressed: redis.clients.jedis.exceptions.JedisConnectionException: Could not get a resource from the pool
                at redis.clients.util.Pool.getResource(Pool.java:53)
                at redis.clients.jedis.JedisPool.getResource(JedisPool.java:226)
                at redis.clients.jedis.JedisPool.getResource(JedisPool.java:16)
                at org.nuxeo.ecm.core.redis.RedisPoolExecutor.execute(RedisPoolExecutor.java:56)
                at org.nuxeo.ecm.core.redis.RedisFailoverExecutor$1.retry(RedisFailoverExecutor.java:62)
                at org.nuxeo.ecm.core.redis.retry.Retry.retry(Retry.java:64)
                ... 13 more
        Caused by: java.lang.IllegalStateException: Pool not open
                at org.apache.commons.pool2.impl.BaseGenericObjectPool.assertOpen(BaseGenericObjectPool.java:672)
                at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:412)
                at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:363)
                at redis.clients.util.Pool.getResource(Pool.java:49)
                ... 18 more
2020-09-29 11:08:30,132 ERROR [WorkManagerImpl] Uncaught error on thread: Nuxeo-Work-default-1, current work might be lost, WorkManager metrics might be corrupted.
org.nuxeo.ecm.core.api.NuxeoException: Work failed after 0 retries, class=class org.nuxeo.ecm.core.transientstore.TransiantStorageGCWork id=7819805909600.317574901 category=TransiantStorageGCWork title=Transient Store GC
        at org.nuxeo.ecm.core.work.AbstractWork.workFailed(AbstractWork.java:437)
        at org.nuxeo.ecm.core.work.AbstractWork.run(AbstractWork.java:393)
        at org.nuxeo.ecm.core.work.WorkHolder.run(WorkHolder.java:57)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: redis.clients.jedis.exceptions.JedisConnectionException: Cannot reconnect to jedis ..
        at org.nuxeo.ecm.core.redis.RedisFailoverExecutor.executeWithRetryPolicy(RedisFailoverExecutor.java:70)
        at org.nuxeo.ecm.core.redis.RedisFailoverExecutor.execute(RedisFailoverExecutor.java:43)
        at org.nuxeo.ecm.core.redis.contribs.RedisTransientStore.setStorageSize(RedisTransientStore.java:424)
        at org.nuxeo.ecm.core.transientstore.AbstractTransientStore.doGC(AbstractTransientStore.java:255)
        at java.util.HashMap$Values.forEach(HashMap.java:981)
        at org.nuxeo.ecm.core.transientstore.TransientStorageComponent.doGC(TransientStorageComponent.java:74)
        at org.nuxeo.ecm.core.transientstore.TransiantStorageGCWork.work(TransiantStorageGCWork.java:41)
        at org.nuxeo.ecm.core.work.AbstractWork.runWorkWithTransaction(AbstractWork.java:491)
        at org.nuxeo.ecm.core.work.AbstractWork.run(AbstractWork.java:381)
        ... 4 more
Caused by: org.nuxeo.ecm.core.redis.retry.Retry$FailException: Cannot execute block, retry policy failed, check suppressed exception for more infos
        at org.nuxeo.ecm.core.redis.retry.Retry.retry(Retry.java:60)
        at org.nuxeo.ecm.core.redis.RedisFailoverExecutor.executeWithRetryPolicy(RedisFailoverExecutor.java:57)
        ... 12 more
        Suppressed: redis.clients.jedis.exceptions.JedisConnectionException: Could not get a resource from the pool
                at redis.clients.util.Pool.getResource(Pool.java:53)
                at redis.clients.jedis.JedisPool.getResource(JedisPool.java:226)
                at redis.clients.jedis.JedisPool.getResource(JedisPool.java:16)
                at org.nuxeo.ecm.core.redis.RedisPoolExecutor.execute(RedisPoolExecutor.java:56)
                at org.nuxeo.ecm.core.redis.RedisFailoverExecutor$1.retry(RedisFailoverExecutor.java:62)
                at org.nuxeo.ecm.core.redis.retry.Retry.retry(Retry.java:64)
                ... 13 more
        Caused by: java.lang.IllegalStateException: Pool not open
                at org.apache.commons.pool2.impl.BaseGenericObjectPool.assertOpen(BaseGenericObjectPool.java:672)
                at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:412)
                at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:363)
                at redis.clients.util.Pool.getResource(Pool.java:49)
                ... 18 more
        Suppressed: redis.clients.jedis.exceptions.JedisConnectionException: Could not get a resource from the pool
                at redis.clients.util.Pool.getResource(Pool.java:53)
                at redis.clients.jedis.JedisPool.getResource(JedisPool.java:226)
                at redis.clients.jedis.JedisPool.getResource(JedisPool.java:16)
                at org.nuxeo.ecm.core.redis.RedisPoolExecutor.execute(RedisPoolExecutor.java:56)
                at org.nuxeo.ecm.core.redis.RedisFailoverExecutor$1.retry(RedisFailoverExecutor.java:62)
                at org.nuxeo.ecm.core.redis.retry.Retry.retry(Retry.java:64)
                ... 13 more
        Caused by: java.lang.IllegalStateException: Pool not open
                at org.apache.commons.pool2.impl.BaseGenericObjectPool.assertOpen(BaseGenericObjectPool.java:672)
                at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:412)
                at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:363)
                at redis.clients.util.Pool.getResource(Pool.java:49)
                ... 18 more
        Suppressed: redis.clients.jedis.exceptions.JedisConnectionException: Could not get a resource from the pool
                at redis.clients.util.Pool.getResource(Pool.java:53)
                at redis.clients.jedis.JedisPool.getResource(JedisPool.java:226)
                at redis.clients.jedis.JedisPool.getResource(JedisPool.java:16)
                at org.nuxeo.ecm.core.redis.RedisPoolExecutor.execute(RedisPoolExecutor.java:56)
                at org.nuxeo.ecm.core.redis.RedisFailoverExecutor$1.retry(RedisFailoverExecutor.java:62)
                at org.nuxeo.ecm.core.redis.retry.Retry.retry(Retry.java:64)
                ... 13 more
        Caused by: java.lang.IllegalStateException: Pool not open
                at org.apache.commons.pool2.impl.BaseGenericObjectPool.assertOpen(BaseGenericObjectPool.java:672)
                at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:412)
                at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:363)
                at redis.clients.util.Pool.getResource(Pool.java:49)
                ... 18 more
        Suppressed: redis.clients.jedis.exceptions.JedisConnectionException: Could not get a resource from the pool
                at redis.clients.util.Pool.getResource(Pool.java:53)
                at redis.clients.jedis.JedisPool.getResource(JedisPool.java:226)
                at redis.clients.jedis.JedisPool.getResource(JedisPool.java:16)
                at org.nuxeo.ecm.core.redis.RedisPoolExecutor.execute(RedisPoolExecutor.java:56)
                at org.nuxeo.ecm.core.redis.RedisFailoverExecutor$1.retry(RedisFailoverExecutor.java:62)
                at org.nuxeo.ecm.core.redis.retry.Retry.retry(Retry.java:64)
                ... 13 more
        Caused by: java.lang.IllegalStateException: Pool not open
                at org.apache.commons.pool2.impl.BaseGenericObjectPool.assertOpen(BaseGenericObjectPool.java:672)
                at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:412)
                at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:363)
                at redis.clients.util.Pool.getResource(Pool.java:49)
                ... 18 more
        Suppressed: redis.clients.jedis.exceptions.JedisConnectionException: Could not get a resource from the pool
                at redis.clients.util.Pool.getResource(Pool.java:53)
                at redis.clients.jedis.JedisPool.getResource(JedisPool.java:226)
                at redis.clients.jedis.JedisPool.getResource(JedisPool.java:16)
                at org.nuxeo.ecm.core.redis.RedisPoolExecutor.execute(RedisPoolExecutor.java:56)
                at org.nuxeo.ecm.core.redis.RedisFailoverExecutor$1.retry(RedisFailoverExecutor.java:62)
                at org.nuxeo.ecm.core.redis.retry.Retry.retry(Retry.java:64)
                ... 13 more
        Caused by: java.lang.IllegalStateException: Pool not open
                at org.apache.commons.pool2.impl.BaseGenericObjectPool.assertOpen(BaseGenericObjectPool.java:672)
                at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:412)
                at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:363)
                at redis.clients.util.Pool.getResource(Pool.java:49)
                ... 18 more
        Suppressed: redis.clients.jedis.exceptions.JedisConnectionException: Could not get a resource from the pool
                at redis.clients.util.Pool.getResource(Pool.java:53)
                at redis.clients.jedis.JedisPool.getResource(JedisPool.java:226)
                at redis.clients.jedis.JedisPool.getResource(JedisPool.java:16)
                at org.nuxeo.ecm.core.redis.RedisPoolExecutor.execute(RedisPoolExecutor.java:56)
                at org.nuxeo.ecm.core.redis.RedisFailoverExecutor$1.retry(RedisFailoverExecutor.java:62)
                at org.nuxeo.ecm.core.redis.retry.Retry.retry(Retry.java:64)
                ... 13 more
        Caused by: java.lang.IllegalStateException: Pool not open
                at org.apache.commons.pool2.impl.BaseGenericObjectPool.assertOpen(BaseGenericObjectPool.java:672)
                at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:412)
                at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:363)
                at redis.clients.util.Pool.getResource(Pool.java:49)
                ... 18 more
        Suppressed: redis.clients.jedis.exceptions.JedisConnectionException: Could not get a resource from the pool
                at redis.clients.util.Pool.getResource(Pool.java:53)
                at redis.clients.jedis.JedisPool.getResource(JedisPool.java:226)
                at redis.clients.jedis.JedisPool.getResource(JedisPool.java:16)
                at org.nuxeo.ecm.core.redis.RedisPoolExecutor.execute(RedisPoolExecutor.java:56)
                at org.nuxeo.ecm.core.redis.RedisFailoverExecutor$1.retry(RedisFailoverExecutor.java:62)
                at org.nuxeo.ecm.core.redis.retry.Retry.retry(Retry.java:64)
                ... 13 more
        Caused by: java.lang.IllegalStateException: Pool not open
                at org.apache.commons.pool2.impl.BaseGenericObjectPool.assertOpen(BaseGenericObjectPool.java:672)
                at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:412)
                at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:363)
                at redis.clients.util.Pool.getResource(Pool.java:49)
                ... 18 more
        Suppressed: redis.clients.jedis.exceptions.JedisConnectionException: Could not get a resource from the pool
                at redis.clients.util.Pool.getResource(Pool.java:53)
                at redis.clients.jedis.JedisPool.getResource(JedisPool.java:226)
                at redis.clients.jedis.JedisPool.getResource(JedisPool.java:16)
                at org.nuxeo.ecm.core.redis.RedisPoolExecutor.execute(RedisPoolExecutor.java:56)
                at org.nuxeo.ecm.core.redis.RedisFailoverExecutor$1.retry(RedisFailoverExecutor.java:62)
                at org.nuxeo.ecm.core.redis.retry.Retry.retry(Retry.java:64)
                ... 13 more
        Caused by: java.lang.IllegalStateException: Pool not open
                at org.apache.commons.pool2.impl.BaseGenericObjectPool.assertOpen(BaseGenericObjectPool.java:672)
                at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:412)
                at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:363)
                at redis.clients.util.Pool.getResource(Pool.java:49)
                ... 18 more
2020-09-29 11:15:00,028 WARN  [SimpleDelay] pausing for 2000 ms
2020-09-29 11:15:02,040 WARN  [SimpleDelay] pausing for 4000 ms
2020-09-29 11:15:06,051 WARN  [SimpleDelay] pausing for 8000 ms
2020-09-29 11:15:14,052 WARN  [SimpleDelay] pausing for 16000 ms
2020-09-29 11:15:30,063 WARN  [SimpleDelay] pausing for 32000 ms
2020-09-29 11:16:02,085 WARN  [SimpleDelay] pausing for 64000 ms
2020-09-29 11:17:06,085 WARN  [SimpleDelay] pausing for 128000 ms
2020-09-29 11:19:14,108 WARN  [SimpleDelay] pausing for 256000 ms
2020-09-29 11:23:30,114 ERROR [AbstractWork] Exception during work: TransiantStorageGCWork(, Progress(?%, ?/0), null)
redis.clients.jedis.exceptions.JedisConnectionException: Cannot reconnect to jedis ..

Analysis shows that https://github.com/nuxeo/nuxeo/blob/9.10/nuxeo-core/nuxeo-core-cache/src/main/java/org/nuxeo/ecm/core/transientstore/TransientStorageComponent.java#L74 expect the stores not to change in the JVM lifetime, which seems wrong at least with HotReload playing.

Attachments

Activity

People

Assignee:

Unassigned

Reporter:

Patrick Abgrall

Participants:

Patrick Abgrall, Thierry Martins

Votes:

0 Vote for this issue

Watchers:

2 Start watching this issue

Dates

Created:

2020-09-29 09:51

Updated:

2020-09-30 13:30

Resolved:

2020-09-30 13:30