Uploaded image for project: 'Nuxeo Platform'
  1. Nuxeo Platform
  2. NXP-27863

Fix possible race condition in Avro schema initialization

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 10.10
    • Fix Version/s: 10.10-HF14, 11.1
    • Component/s: Streams

      Description

      Reported problem:

      1. Start a fresh Nuxeo it hangs, no error, nuxeoctl gives up (too long), server forcibily stopped
      2. When restarting there is an error:

      2019-07-17T17:53:25,406 ERROR [main] [org.nuxeo.runtime.model.ComponentManager] Component service:org.nuxeo.runtime.avro notification of application started failed: Invalid schema file: /var/lib/nuxeo/data/avro/BulkStatus-0x182AC639648607C7.avsc java.lang.IllegalArgumentException: Invalid schema file: /var/lib/nuxeo/data/avro/BulkStatus-0x182AC639648607C7.avsc at ...
      org.nuxeo.lib.stream.codec.FileAvroSchemaStore.loadSchema(FileAvroSchemaStore.java:81) ~[nuxeo-stream-10.10-HF10.jar:?] at 
      org.nuxeo.lib.stream.codec.FileAvroSchemaStore.loadSchemas(FileAvroSchemaStore.java:67) ~[nuxeo-stream-10.10-HF10.jar:?] at org.nuxeo.lib.stream.codec.FileAvroSchemaStore.<init>(FileAvroSchemaStore.java:55) ~[nuxeo-stream-10.10-HF10.jar:?] at org.nuxeo.runtime.avro.AvroServiceImpl.<init>(AvroServiceImpl.java:73) ~[nuxeo-runtime-stream-10.10-HF08.jar:?] at 
      

      The invalid Avro schema file exists but it is empty thus invalid.

      This prevents to start Nuxeo properly

      ======================================================================
      = Component Loading Status: Pending: 0 / Missing: 0 / Unstarted: 2 / Total: 818
        - service:org.nuxeo.runtime.stream.service
        - service:org.nuxeo.runtime.avro
      ======================================================================
      

      Without Avro or Bulk service other NPE errors are triggered, like:

          Caused by: java.lang.NullPointerExceptionCaused by: java.lang.NullPointerException at org.nuxeo.runtime.avro.AvroComponent.getAdapter(AvroComponent.java:56) ~[nuxeo-runtime-stream-10.10-HF08.jar:?]
      

      This can be fixed by removing the empty file (/var/lib/nuxeo/data/avro/BulkStatus-0x182AC639648607C7.avsc) and restarting.

      Without a thread dump, it is hard to be sure but It is possible that there is a race condition when creating the Avro Schema files, if multiple threads try to create the same schema file it could result in a deadlock with an empty file.

        Attachments

          Issue Links

            Activity

              People

              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0 minutes
                  0m
                  Logged:
                  Time Spent - 3 hours
                  3h

                    PagerDuty

                    Error rendering 'com.pagerduty.jira-server-plugin:PagerDuty'. Please contact your Jira administrators.