Fix concurrency issue with MongoDB leading to inconsistent world

What
The concurrency issue described here creates the condition for NXP-30000: a document can reference in ecm:proxyIds ids of formerly existing proxies that are now deleted.
The request is to avoid the concurrency issue as far as can be to keep a consistent world.

How to reproduce

• Take a machine with many CPUs and COREs (e.g. 12 COREs)
• Start a MongoDB instance and allocate e.g. 8 CORES or CPU to MongoDB to maximize concurrency possibilities, e.g.

    docker run --name mongo36 --cpus 8 -v /<DIRECTORY>/<PATH>/d/mongodb-data:/data/db -p 27017:27017 -d mongo:3.6
    

• Start a hotfixed Nuxeo instance with WebUI and JSF and DAM and SHOWCASE-content and target mongo as a database
• Create 50 sections
• Publish the PDF /default-domain/workspaces/Sample%20Content/PDF%20and%20Office%20Documents/Sample%20PDF%20File.pdf in the 50 sections
  This will thus create 50 proxies targeting 1 single document
• Run the following command to delete the 50 proxies with maximum velocity and concurrency

    curl -u Administrator:Administrator -X GET "http://<NUXEO>:8080/nuxeo/api/v1/search/lang/NXQL/execute?query=SELECT%20*%20FROM%20Document%20WHERE%20ecm%3AisProxy%3D1" -H "properties: common" -H "Content-Type: application/json" -H "Accept: application/json"|jq ".|.entries[] | (.uid)"  |sed -e 's/"//g'|while read i ; do
          # echo $i & 
          curl -u Administrator:Administrator -X DELETE "http://<NUXEO>:8080/nuxeo/api/v1/id/$i" &
    done
    

• You will get many error statements about ConcurrentUpdateException, and only some few proxies will effectively be deleted
• If you retrieve the ID of the original version target document (here 20b85394-168c-4cc8-8ba3-395b53879937):
  • First attempt to delete it:

    	 curl -u Administrator:Administrator -X DELETE "http://<NUXEO>:8080/nuxeo/api/v1/id/20b85394-168c-4cc8-8ba3-395b53879937"
    	 

    you will get a 403 due to a proxy targeting the document (this is the MAIN consequence of the issue to solve)

– When going in MongoDB to look at the target document:

     db=db.getSiblingDB("nuxeo")
     coll=db.getCollection("default")
     coll2 = coll.find({"ecm:id":"20b85394-168c-4cc8-8ba3-395b53879937"}).toArray()
	 

When adapting the output to be "jq-compatible", you will get an output such as in the attached file mongo_output_for_20b85394-168c-4cc8-8ba3-395b53879937

– When analyzing this file with the script:

cat mongo_output_for_20b85394-168c-4cc8-8ba3-395b53879937 |jq ".|(.\"ecm:proxyIds\")" |sed -e 's/"//g' -e 's/,//' -e "s/.*\[//" -e 's/]//'|while read i ; do
        if [ "$i" != "" ]; then
                echo -n "Proxy Id : "$i" "
                curl -s -u Administrator:Administrator -X GET "http://<NUXEO>:8080/nuxeo/api/v1/id/$i" | grep "status.:404"
                echo
        fi
done

you will see that some referenced proxy ids do not match any existing proxy, e.g.

	 [...]
	 Proxy Id : a44ede3f-7bdc-46e1-8663-4d7e7182915b
     Proxy Id : f2d0c454-79e7-4a6a-8481-e68d59c946a3 {"entity-type":"exception","status":404,"message":"f2d0c454-79e7-4a6a-8481-e68d59c946a3"}
	 [...]
     

Consequence
As at least one proxy is still referenced in the target document, the target document cannot be deleted.

REQUEST
Avoid concurrency issues in MongoDB to keep the world consistent/coherent.
As such when running the concurrent requests

  curl -u Administrator:Administrator -X GET "http://<NUXEO>:8080/nuxeo/api/v1/search/lang/NXQL/execute?query=SELECT%20*%20FROM%20Document%20WHERE%20ecm%3AisProxy%3D1" -H "properties: common" -H "Content-Type: application/json" -H "Accept: application/json"|jq ".|.entries[] | (.uid)"  |sed -e 's/"//g'|while read i ; do
        # echo $i & 
        curl -u Administrator:Administrator -X DELETE "http://<NUXEO>:8080/nuxeo/api/v1/id/$i" &
  done

there should be no ConcurrentUpdateException raised at any time.