-
Type:
Improvement
-
Status: Resolved
-
Priority:
Minor
-
Resolution: Fixed
-
Affects Version/s: None
-
Component/s: Redis
When using Redis (without Sentinel), there is no retry logic setup,
so when a connection error happens, worker thread will raise uncaught error like:
ERROR [Nuxeo-Work-elasticSearchIndexing-2] [org.nuxeo.ecm.core.work.WorkManagerImpl] Uncaught error on thread Nuxeo-Work-elasticSearchIndexing-2 redis.clients.jedis.exceptions.JedisConnectionException: Unexpected end of stream. at redis.clients.util.RedisInputStream.ensureFill(RedisInputStream.java:198) at redis.clients.util.RedisInputStream.readByte(RedisInputStream.java:40) at redis.clients.jedis.Protocol.process(Protocol.java:132) at redis.clients.jedis.Protocol.read(Protocol.java:196) at redis.clients.jedis.Connection.readProtocolWithCheckingBroken(Connection.java:288) at redis.clients.jedis.Connection.getBinaryBulkReply(Connection.java:207) at redis.clients.jedis.BinaryJedis.rpop(BinaryJedis.java:1161) at org.nuxeo.ecm.core.redis.contribs.RedisWorkQueuing$16.call(RedisWorkQueuing.java:805) at org.nuxeo.ecm.core.redis.contribs.RedisWorkQueuing$16.call(RedisWorkQueuing.java:800) at org.nuxeo.ecm.core.redis.RedisPoolExecutor.execute(RedisPoolExecutor.java:31) at org.nuxeo.ecm.core.redis.contribs.RedisWorkQueuing.removeScheduledWork(RedisWorkQueuing.java:800) at org.nuxeo.ecm.core.redis.contribs.RedisBlockingQueue.pollElement(RedisBlockingQueue.java:99) at org.nuxeo.ecm.core.work.NuxeoBlockingQueue.poll(NuxeoBlockingQueue.java:99) at org.nuxeo.ecm.core.redis.contribs.RedisBlockingQueue.poll(RedisBlockingQueue.java:73) at org.nuxeo.ecm.core.redis.contribs.RedisBlockingQueue.take(RedisBlockingQueue.java:57) at org.nuxeo.ecm.core.redis.contribs.RedisBlockingQueue.take(RedisBlockingQueue.java:36) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745)
When using Sentinel, Nuxeo uses a failover executor with an exponential delay policy: (1ms, 2ms, 4ms, 8ms, 16ms, 32ms .. until the timeout is reached 5min by default).
We should use the same failover executor by default for Redis.
–
Redis communcation is now more reliable by using a retry logic policy in case of unreachable access.
