Uploaded image for project: 'Nuxeo Platform'
  1. Nuxeo Platform
  2. NXP-22574

Fix JVM shutdown during MongoDB replica set failover

    XMLWordPrintable

    Details

      Description

      If the primary member of a MongoDB replica set is not available any more and change, the following exception is logged

      ERROR [CheckinTask] Node nereid2.in.nuxeo.com1497959511117 could not check-in: Prematurely reached end of stream
      com.mongodb.MongoSocketReadException: Prematurely reached end of stream
      	at com.mongodb.connection.SocketStream.read(SocketStream.java:88)
      	at com.mongodb.connection.InternalStreamConnection.receiveResponseBuffers(InternalStreamConnection.java:492)
      	at com.mongodb.connection.InternalStreamConnection.receiveMessage(InternalStreamConnection.java:222)
      	at com.mongodb.connection.UsageTrackingInternalConnection.receiveMessage(UsageTrackingInternalConnection.java:105)
      	at com.mongodb.connection.DefaultConnectionPool$PooledConnection.receiveMessage(DefaultConnectionPool.java:438)
      	at com.mongodb.connection.WriteCommandProtocol.receiveMessage(WriteCommandProtocol.java:262)
      	at com.mongodb.connection.WriteCommandProtocol.execute(WriteCommandProtocol.java:104)
      	at com.mongodb.connection.UpdateCommandProtocol.execute(UpdateCommandProtocol.java:64)
      	at com.mongodb.connection.UpdateCommandProtocol.execute(UpdateCommandProtocol.java:37)
      	at com.mongodb.connection.DefaultServer$DefaultServerProtocolExecutor.execute(DefaultServer.java:168)
      	at com.mongodb.connection.DefaultServerConnection.executeProtocol(DefaultServerConnection.java:289)
      	at com.mongodb.connection.DefaultServerConnection.updateCommand(DefaultServerConnection.java:143)
      	at com.mongodb.operation.MixedBulkWriteOperation$Run$3.executeWriteCommandProtocol(MixedBulkWriteOperation.java:481)
      	at com.mongodb.operation.MixedBulkWriteOperation$Run$RunExecutor.execute(MixedBulkWriteOperation.java:647)
      	at com.mongodb.operation.MixedBulkWriteOperation$Run.execute(MixedBulkWriteOperation.java:400)
      	at com.mongodb.operation.MixedBulkWriteOperation$1.call(MixedBulkWriteOperation.java:180)
      	at com.mongodb.operation.MixedBulkWriteOperation$1.call(MixedBulkWriteOperation.java:169)
      	at com.mongodb.operation.OperationHelper.withConnectionSource(OperationHelper.java:232)
      	at com.mongodb.operation.OperationHelper.withConnection(OperationHelper.java:223)
      	at com.mongodb.operation.MixedBulkWriteOperation.execute(MixedBulkWriteOperation.java:169)
      	at com.mongodb.operation.MixedBulkWriteOperation.execute(MixedBulkWriteOperation.java:75)
      	at com.mongodb.Mongo.execute(Mongo.java:827)
      	at com.mongodb.Mongo$2.execute(Mongo.java:810)
      	at com.mongodb.MongoCollectionImpl.executeSingleWriteRequest(MongoCollectionImpl.java:515)
      	at com.mongodb.MongoCollectionImpl.update(MongoCollectionImpl.java:508)
      	at com.mongodb.MongoCollectionImpl.updateOne(MongoCollectionImpl.java:355)
      	at com.novemberain.quartz.mongodb.dao.SchedulerDao.checkIn(SchedulerDao.java:71)
      	at com.novemberain.quartz.mongodb.cluster.CheckinTask.run(CheckinTask.java:46)
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
      	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
      	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
      	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      	at java.lang.Thread.run(Thread.java:745)
      

      and the JVM stops because of the following code from MongoDB Quartz driver

      CheckinTask.java
          private static final Runnable DEFAULT_ERROR_HANDLER = new Runnable() {
              @Override
              public void run() {
                  //TODO Is there a way to stop only Quartz?
                  System.exit(1);
              }
          };
      

      This code must be patched to at least retry and to prevent from shutting down Nuxeo process.

      Same issue: https://github.com/michaelklishin/quartz-mongodb/issues/147

        Attachments

          Issue Links

            Activity

              People

              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0 minutes
                  0m
                  Logged:
                  Time Spent - 1 hour
                  1h

                    PagerDuty

                    Error rendering 'com.pagerduty.jira-server-plugin:PagerDuty'. Please contact your Jira administrators.