Uploaded image for project: 'Nuxeo Platform'
  1. Nuxeo Platform
  2. NXP-16241

Prevent race condition on cluster quartz init

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 5.8, 6.0
    • Fix Version/s: 5.8.0-HF29, 6.0-HF03, 7.1
    • Component/s: Core

      Description

      If a Nuxeo node starts the scheduler, and therefore Quartz, while another Nuxeo node has a Quartz trigger executing (or even the same node in the case of the MisfireHandler thread relaunching late triggers), there scheduler init must not be able to delete a job at the same time that a job executes.

      To do that there should be proper lock-based exclusion, which relies on a SQL transaction being used. This requires:

      • dontSetAutoCommitFalse = false
      • acquireTriggersWithinLock = true

      Otherwise the following error may occur (but note that it's benign, has it happens in a trigger thread, and will get retried at the next trigger occurence):

      ERROR [org.quartz.core.ErrorLogger] An error occured while firing trigger 'nuxeo.workManagerCleanup'
      org.quartz.JobPersistenceException: Couldn't store trigger 'workManagerCleanup' for 'workManagerCleanup' job:ERROR: insert or update on table "qrtz_triggers" violates foreign key constraint "qrtz_triggers_job_name_fkey"
        Detail: Key (job_name, job_group)=(workManagerCleanup, nuxeo) is not present in table "qrtz_job_details". [See nested exception: org.postgresql.util.PSQLException: ERROR: insert or update on table "qrtz_triggers" violates foreign key constraint "qrtz_triggers_job_name_fkey"
        Detail: Key (job_name, job_group)=(workManagerCleanup, nuxeo) is not present in table "qrtz_job_details".]
              at org.quartz.impl.jdbcjobstore.JobStoreSupport.storeTrigger(JobStoreSupport.java:1270)
              at org.quartz.impl.jdbcjobstore.JobStoreSupport.triggerFired(JobStoreSupport.java:2961)
              at org.quartz.impl.jdbcjobstore.JobStoreSupport$38.execute(JobStoreSupport.java:2871)
              at org.quartz.impl.jdbcjobstore.JobStoreSupport.executeInNonManagedTXLock(JobStoreSupport.java:3788)
              at org.quartz.impl.jdbcjobstore.JobStoreSupport.triggerFired(JobStoreSupport.java:2865)
              at org.quartz.core.QuartzSchedulerThread.run(QuartzSchedulerThread.java:319)
      Caused by: org.postgresql.util.PSQLException: ERROR: insert or update on table "qrtz_triggers" violates foreign key constraint "qrtz_triggers_job_name_fkey"
        Detail: Key (job_name, job_group)=(workManagerCleanup, nuxeo) is not present in table "qrtz_job_details".
              at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2157)
              at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1886)
              at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:255)
              at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:555)
              at org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:417)
              at org.postgresql.jdbc2.AbstractJdbc2Statement.executeUpdate(AbstractJdbc2Statement.java:363)
              at org.apache.tomcat.dbcp.dbcp.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:105)
              at org.apache.tomcat.dbcp.dbcp.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:105)
              at org.quartz.impl.jdbcjobstore.StdJDBCDelegate.insertTrigger(StdJDBCDelegate.java:1105)
              at org.quartz.impl.jdbcjobstore.JobStoreSupport.storeTrigger(JobStoreSupport.java:1258)
              ... 5 more
      

        Attachments

          Activity

            People

            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: