[NXP-30891] Possible contention on OAuth when account has many tokens - Nuxeo Issue Tracker

XML

Word

Printable

Details

Type: Bug
Status: Resolved
Priority: Minor
Resolution: Fixed
Affects Version/s: 10.10, 2021
Fix Version/s: 10.10-HF59, 2021.17
Component/s: OAuth

Release Notes Summary:
Mongodb directory entries are now ordered efficiently by date with the DocumentModelComparator performing efficient sorting on date time properties
Tags:
Team:
PLATFORM
Sprint:
nxplatform #56
Story Points:
3

Description

When an account has hundred of OAuth tokens,
the code to sort them by date is very ineffective and creates contention:

One thread is sorting

"http-nio-0.0.0.0-8080-exec-87" #1078 daemon prio=5 os_prio=0 tid=0x00007f1a3c057000 nid=0x6e4 runnable [0x00007f199a1e5000]
   java.lang.Thread.State: RUNNABLE
	at java.text.Collator.getStrength(Collator.java:347)
	- locked <0x00000003c890d420> (a java.text.RuleBasedCollator)
	at java.text.CollationElementIterator.strengthOrder(CollationElementIterator.java:393)
	at java.text.CollationElementIterator.next(CollationElementIterator.java:261)
	at java.text.RuleBasedCollator.compare(RuleBasedCollator.java:404)
	- locked <0x00000003c890d420> (a java.text.RuleBasedCollator)
	at org.nuxeo.ecm.core.api.DocumentModelComparator.compare(DocumentModelComparator.java:88)
	at org.nuxeo.ecm.core.api.DocumentModelComparator.compare(DocumentModelComparator.java:112)
	at org.nuxeo.ecm.core.api.DocumentModelComparator.compare(DocumentModelComparator.java:36)
	at java.util.Arrays.mergeSort(Arrays.java:1567)
	at java.util.Arrays.mergeSort(Arrays.java:1556)
	at java.util.Arrays.mergeSort(Arrays.java:1555)
	at java.util.Arrays.mergeSort(Arrays.java:1555)
	at java.util.Arrays.mergeSort(Arrays.java:1556)
	at java.util.Arrays.legacyMergeSort(Arrays.java:1523)
	at java.util.Arrays.sort(Arrays.java:1510)
	at java.util.ArrayList.sort(ArrayList.java:1464)
	at org.nuxeo.ecm.directory.AbstractDirectory.orderEntries(AbstractDirectory.java:268)
	at org.nuxeo.directory.mongodb.MongoDBSession.doQuery(MongoDBSession.java:368)
	at org.nuxeo.directory.mongodb.MongoDBSession.query(MongoDBSession.java:298)
	at org.nuxeo.ecm.directory.BaseSession.query(BaseSession.java:562)
	at org.nuxeo.ecm.directory.BaseSession.query(BaseSession.java:556)
	at org.nuxeo.ecm.platform.oauth2.tokens.OAuth2TokenStore.lambda$query$3(OAuth2TokenStore.java:288)
	at org.nuxeo.ecm.platform.oauth2.tokens.OAuth2TokenStore$$Lambda$835/841526078.get(Unknown Source)
	at org.nuxeo.runtime.api.Framework.doPrivileged(Framework.java:293)
	at org.nuxeo.ecm.platform.oauth2.tokens.OAuth2TokenStore.query(OAuth2TokenStore.java:285)
	at org.nuxeo.ecm.platform.oauth2.tokens.OAuth2TokenStore.getToken(OAuth2TokenStore.java:264)
	at org.nuxeo.ecm.platform.oauth2.NuxeoOAuth2Servlet.getAndSendToken(wNuxeoOAuth2Servlet.java:384)
	at org.nuxeo.ecm.platform.oauth2.NuxeoOAuth2Servlet.doPostToken(NuxeoOAuth2Servlet.java:373)
	at org.nuxeo.ecm.platform.oauth2.NuxeoOAuth2Servlet.doPost(NuxeoOAuth2Servlet.java:128)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:681)

many are blocked on the synchronized code:

"http-nio-0.0.0.0-8080-exec-90" #1081 daemon prio=5 os_prio=0 tid=0x00007f1a3c006800 nid=0x6e7 waiting for monitor entry [0x00007f19992d7000]
   java.lang.Thread.State: BLOCKED (on object monitor)
	at java.text.RuleBasedCollator.compare(RuleBasedCollator.java:356)
	- waiting to lock <0x00000003c890d420> (a java.text.RuleBasedCollator)
	at org.nuxeo.ecm.core.api.DocumentModelComparator.compare(DocumentModelComparator.java:88)
	at org.nuxeo.ecm.core.api.DocumentModelComparator.compare(DocumentModelComparator.java:112)
	at org.nuxeo.ecm.core.api.DocumentModelComparator.compare(DocumentModelComparator.java:36)
	at java.util.Arrays.mergeSort(Arrays.java:1567)
	at java.util.Arrays.mergeSort(Arrays.java:1556)
	at java.util.Arrays.legacyMergeSort(Arrays.java:1523)
	at java.util.Arrays.sort(Arrays.java:1510)
	at java.util.ArrayList.sort(ArrayList.java:1464)
	at org.nuxeo.ecm.directory.AbstractDirectory.orderEntries(AbstractDirectory.java:268)
	at org.nuxeo.directory.mongodb.MongoDBSession.doQuery(MongoDBSession.java:368)
	at org.nuxeo.directory.mongodb.MongoDBSession.query(MongoDBSession.java:298)
	at org.nuxeo.ecm.directory.BaseSession.query(BaseSession.java:562)
	at org.nuxeo.ecm.directory.BaseSession.query(BaseSession.java:556)
	at org.nuxeo.ecm.platform.oauth2.tokens.OAuth2TokenStore.lambda$query$3(OAuth2TokenStore.java:288)
	at org.nuxeo.ecm.platform.oauth2.tokens.OAuth2TokenStore$$Lambda$835/841526078.get(Unknown Source)
	at org.nuxeo.runtime.api.Framework.doPrivileged(Framework.java:293)
	at org.nuxeo.ecm.platform.oauth2.tokens.OAuth2TokenStore.query(OAuth2TokenStore.java:285)
	at org.nuxeo.ecm.platform.oauth2.tokens.OAuth2TokenStore.getToken(OAuth2TokenStore.java:264)
	at org.nuxeo.ecm.platform.oauth2.NuxeoOAuth2Servlet.getAndSendToken(NuxeoOAuth2Servlet.java:384)
	at org.nuxeo.ecm.platform.oauth2.NuxeoOAuth2Servlet.doPostToken(NuxeoOAuth2Servlet.java:373)
	at org.nuxeo.ecm.platform.oauth2.NuxeoOAuth2Servlet.doPost(NuxeoOAuth2Servlet.java:128)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:681)

There are 2 problems:

RuleBasedCollator should be avoided because it uses synchronized code (should use a thread local like in https://stackoverflow.com/questions/36135914/how-to-avoid-thread-blockage-during-concurrent-sorting-with-a-java-text-collator/36136527#36136527)
OAuth2TokenStore is retrieving all tokens just to take the most recent

Temporary workaround: try to limit the number of tokens per clientId/userId.

Attachments

Activity

People

Assignee:

Guillaume Renard

Reporter:

Benoit Delbosc

Participants:

Benoit Delbosc, Guillaume Renard, Jenkins, Support Tech User

Votes:

1 Vote for this issue

Watchers:

7 Start watching this issue

Dates

Created:

2022-02-21 11:24

Updated:

2022-08-11 17:01

Resolved:

2022-03-08 09:56