Uploaded image for project: 'Nuxeo Platform'
  1. Nuxeo Platform
  2. NXP-30001

Fix concurrency issue with MongoDB leading to inconsistent world

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Duplicate
    • Affects Version/s: 10.10
    • Fix Version/s: None
    • Component/s: Core MongoDB
    • Tags:
    • Backlog priority:
      800

      Description

      What
      The concurrency issue described here creates the condition for NXP-30000: a document can reference in ecm:proxyIds ids of formerly existing proxies that are now deleted.
      The request is to avoid the concurrency issue as far as can be to keep a consistent world.

      How to reproduce

      • Take a machine with many CPUs and COREs (e.g. 12 COREs)
      • Start a MongoDB instance and allocate e.g. 8 CORES or CPU to MongoDB to maximize concurrency possibilities, e.g.
          docker run --name mongo36 --cpus 8 -v /<DIRECTORY>/<PATH>/d/mongodb-data:/data/db -p 27017:27017 -d mongo:3.6
          
      • Start a hotfixed Nuxeo instance with WebUI and JSF and DAM and SHOWCASE-content and target mongo as a database
      • Create 50 sections
      • Publish the PDF /default-domain/workspaces/Sample%20Content/PDF%20and%20Office%20Documents/Sample%20PDF%20File.pdf in the 50 sections
        This will thus create 50 proxies targeting 1 single document
      • Run the following command to delete the 50 proxies with maximum velocity and concurrency
          curl -u Administrator:Administrator -X GET "http://<NUXEO>:8080/nuxeo/api/v1/search/lang/NXQL/execute?query=SELECT%20*%20FROM%20Document%20WHERE%20ecm%3AisProxy%3D1" -H "properties: common" -H "Content-Type: application/json" -H "Accept: application/json"|jq ".|.entries[] | (.uid)"  |sed -e 's/"//g'|while read i ; do
                # echo $i & 
                curl -u Administrator:Administrator -X DELETE "http://<NUXEO>:8080/nuxeo/api/v1/id/$i" &
          done
          
      • You will get many error statements about ConcurrentUpdateException, and only some few proxies will effectively be deleted
      • If you retrieve the ID of the original version target document (here 20b85394-168c-4cc8-8ba3-395b53879937):
        • First attempt to delete it:
          	 curl -u Administrator:Administrator -X DELETE "http://<NUXEO>:8080/nuxeo/api/v1/id/20b85394-168c-4cc8-8ba3-395b53879937"
          	 

          you will get a 403 due to a proxy targeting the document (this is the MAIN consequence of the issue to solve)

      – When going in MongoDB to look at the target document:

           db=db.getSiblingDB("nuxeo")
           coll=db.getCollection("default")
           coll2 = coll.find({"ecm:id":"20b85394-168c-4cc8-8ba3-395b53879937"}).toArray()
      	 

      When adapting the output to be "jq-compatible", you will get an output such as in the attached file mongo_output_for_20b85394-168c-4cc8-8ba3-395b53879937

      – When analyzing this file with the script:

      cat mongo_output_for_20b85394-168c-4cc8-8ba3-395b53879937 |jq ".|(.\"ecm:proxyIds\")" |sed -e 's/"//g' -e 's/,//' -e "s/.*\[//" -e 's/]//'|while read i ; do
              if [ "$i" != "" ]; then
                      echo -n "Proxy Id : "$i" "
                      curl -s -u Administrator:Administrator -X GET "http://<NUXEO>:8080/nuxeo/api/v1/id/$i" | grep "status.:404"
                      echo
              fi
      done
      

      you will see that some referenced proxy ids do not match any existing proxy, e.g.

      	 [...]
      	 Proxy Id : a44ede3f-7bdc-46e1-8663-4d7e7182915b
           Proxy Id : f2d0c454-79e7-4a6a-8481-e68d59c946a3 {"entity-type":"exception","status":404,"message":"f2d0c454-79e7-4a6a-8481-e68d59c946a3"}
      	 [...]
           

      Consequence
      As at least one proxy is still referenced in the target document, the target document cannot be deleted.

      REQUEST
      Avoid concurrency issues in MongoDB to keep the world consistent/coherent.
      As such when running the concurrent requests

        curl -u Administrator:Administrator -X GET "http://<NUXEO>:8080/nuxeo/api/v1/search/lang/NXQL/execute?query=SELECT%20*%20FROM%20Document%20WHERE%20ecm%3AisProxy%3D1" -H "properties: common" -H "Content-Type: application/json" -H "Accept: application/json"|jq ".|.entries[] | (.uid)"  |sed -e 's/"//g'|while read i ; do
              # echo $i & 
              curl -u Administrator:Administrator -X DELETE "http://<NUXEO>:8080/nuxeo/api/v1/id/$i" &
        done
      

      there should be no ConcurrentUpdateException raised at any time.

        Attachments

          Issue Links

            Activity

              People

              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  PagerDuty

                  Error rendering 'com.pagerduty.jira-server-plugin:PagerDuty'. Please contact your Jira administrators.