Uploaded image for project: 'Nuxeo Platform'
  1. Nuxeo Platform
  2. NXP-20969

Fix Redis HA mode when Redis nodes are restarted

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 7.10
    • Fix Version/s: 7.10-HF18, 8.10
    • Component/s: Redis

      Description

      When all the nodes in a Redis cluster are restarted while Nuxeo is keeping up, some errors will be raised like

      ERROR [http-bio-10.201.75.17-8080-exec-6] [nuxeo-error-log] redis.clients.jedis.exceptions.JedisDataException: NOSCRIPT No matching script. Please use EVAL.
      at redis.clients.jedis.Protocol.processError(Protocol.java:117)
      at redis.clients.jedis.Protocol.process(Protocol.java:142)
      at redis.clients.jedis.Protocol.read(Protocol.java:196)
      at redis.clients.jedis.Connection.readProtocolWithCheckingBroken(Connection.java:288)
      at redis.clients.jedis.Connection.getOne(Connection.java:270)
      at redis.clients.jedis.BinaryJedis.evalsha(BinaryJedis.java:3001)
      at redis.clients.jedis.BinaryJedis.evalsha(BinaryJedis.java:2994)
      at org.nuxeo.ecm.core.redis.contribs.RedisWorkQueuing$3.call(RedisWorkQueuing.java:513)
      at org.nuxeo.ecm.core.redis.contribs.RedisWorkQueuing$3.call(RedisWorkQueuing.java:509)
      at org.nuxeo.ecm.core.redis.RedisPoolExecutor.execute(RedisPoolExecutor.java:46)
      at org.nuxeo.ecm.core.redis.RedisFailoverExecutor$1.retry(RedisFailoverExecutor.java:39)
      at org.nuxeo.ecm.core.redis.retry.Retry.retry(Retry.java:54)
      at org.nuxeo.ecm.core.redis.RedisFailoverExecutor.execute(RedisFailoverExecutor.java:34)
      at org.nuxeo.ecm.core.redis.contribs.RedisWorkQueuing.addScheduledWork(RedisWorkQueuing.java:509)
      at org.nuxeo.ecm.core.redis.contribs.RedisBlockingQueue.putElement(RedisBlockingQueue.java:124)
      at org.nuxeo.ecm.core.work.NuxeoBlockingQueue.put(NuxeoBlockingQueue.java:85)
      at org.nuxeo.ecm.core.work.NuxeoBlockingQueue.offer(NuxeoBlockingQueue.java:68)
      at org.nuxeo.ecm.core.work.NuxeoBlockingQueue.offer(NuxeoBlockingQueue.java:33)
      at org.nuxeo.ecm.core.redis.contribs.RedisWorkQueuing.workSchedule(RedisWorkQueuing.java:182)
      at org.nuxeo.ecm.core.work.WorkManagerImpl$WorkThreadPoolExecutor.submit(WorkManagerImpl.java:545)
      at org.nuxeo.ecm.core.work.WorkManagerImpl$WorkThreadPoolExecutor.execute(WorkManagerImpl.java:534)
      at org.nuxeo.ecm.core.work.WorkManagerImpl.schedule(WorkManagerImpl.java:706)
      at org.nuxeo.ecm.core.work.WorkManagerImpl.schedule(WorkManagerImpl.java:651)
      at org.nuxeo.ecm.core.event.impl.AsyncEventExecutor.run(AsyncEventExecutor.java:78)
      at org.nuxeo.ecm.core.event.impl.EventServiceImpl.fireEventBundle(EventServiceImpl.java:302)
      at org.nuxeo.ecm.core.event.impl.EventServiceImpl.fireEvent(EventServiceImpl.java:244)
      at org.nuxeo.ecm.platform.ui.web.auth.NuxeoAuthenticationFilter.sendAuthenticationEvent(NuxeoAuthenticationFilter.java:200)
      at org.nuxeo.ecm.platform.ui.web.auth.NuxeoAuthenticationFilter.logAuthenticationAttempt(NuxeoAuthenticationFilter.java:233)
      at org.nuxeo.ecm.platform.ui.web.auth.NuxeoAuthenticationFilter.doAuthenticate(NuxeoAuthenticationFilter.java:291)
      at org.nuxeo.ecm.platform.ui.web.auth.NuxeoAuthenticationFilter.doFilterInternal(NuxeoAuthenticationFilter.java:532)
      at org.nuxeo.ecm.platform.ui.web.auth.service.NuxeoAuthFilterChain.doFilter(NuxeoAuthFilterChain.java:36)
      at org.nuxeo.ecm.platform.ui.web.auth.oauth.NuxeoOAuthFilter.doFilter(NuxeoOAuthFilter.java:120)
      

      This seems to be due to the fact that some scripts are loaded to Redis memory when Nuxeo is initialized and these scripts are lost when a Redis node is restarted.

        Attachments

          Activity

            People

            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Time Tracking

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0 minutes
                0m
                Logged:
                Time Spent - 4 hours
                4h