Discovered since NXP-28579 was merged.
Seems to be long-running tests (recursion, migration, big number of documents, ...).
Next time it happens, let's check the resource usage of the "test-release-postgresql" pod related to the test namespace in the GCP console, for instance https://console.cloud.google.com/kubernetes/pod/us-east1-b/jx-prod/nuxeo-unit-tests-master-304-postgresql/test-release-postgresql-0?project=jx-preprod&container_summary_list_tablesize=10&tab=details&duration=PT6H&service_list_datatablesize=20 (not sure how long the link will be valid...)
What CPU/memory does the pod actually requests on the node?
Does it reach what is configured in the Helm chart?
Are the node resources saturated?
What about the other nodes in the pool? Is the pool big enough?
Might also be a lock issue...
TestSQLBackendNoPathOptims.testRecursiveRemovalBig, historically added for NXP-20211.
Run unit tests / Run postgresql unit tests / org.nuxeo.ecm.core.storage.sql.TestSQLBackendNoPathOptims.testRecursiveRemovalBig org.nuxeo.runtime.transaction.TransactionRuntimeException: Unable to commit: Transaction timeout at org.nuxeo.runtime.transaction.TransactionHelper.commitOrRollbackTransaction(TransactionHelper.java:410) at org.nuxeo.runtime.transaction.TransactionHelper.runInTransaction(TransactionHelper.java:617) at org.nuxeo.ecm.core.storage.sql.jdbc.JDBCMapperConnector.doConnectAndInvoke(JDBCMapperConnector.java:95) at org.nuxeo.ecm.core.storage.sql.jdbc.JDBCMapperConnector.invoke(JDBCMapperConnector.java:84) at com.sun.proxy.$Proxy57.updateReadAcls(Unknown Source) at org.nuxeo.ecm.core.storage.sql.SoftRefCachingMapper.updateReadAcls(SoftRefCachingMapper.java:131) at org.nuxeo.ecm.core.storage.sql.SessionImpl.updateReadAcls(SessionImpl.java:1257) at org.nuxeo.ecm.core.storage.sql.SessionImpl.flush(SessionImpl.java:337) at org.nuxeo.ecm.core.storage.sql.SessionImpl.save(SessionImpl.java:314) at org.nuxeo.ecm.core.storage.sql.TestSQLBackend.testRecursiveRemovalBig(TestSQLBackend.java:410) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.nuxeo.runtime.test.runner.FeaturesRunner$BeforeMethodRunStatement.evaluate(FeaturesRunner.java:308) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.nuxeo.runtime.test.runner.FeaturesRunner$BeforeSetupStatement.evaluate(FeaturesRunner.java:322) at org.nuxeo.runtime.test.runner.FeaturesRunner$AfterMethodRunStatement.evaluate(FeaturesRunner.java:345) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.nuxeo.runtime.test.runner.FeaturesRunner$AfterTeardownStatement.evaluate(FeaturesRunner.java:362) at org.nuxeo.runtime.test.runner.FeaturesRunner$RulesFactory$1.evaluate(FeaturesRunner.java:435) at org.nuxeo.runtime.test.runner.RuntimeDeployment$DeploymentStatement.evaluate(RuntimeDeployment.java:326) at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306) at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63) at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329) at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293) at org.nuxeo.runtime.test.runner.FeaturesRunner$AfterClassStatement.evaluate(FeaturesRunner.java:251) at org.nuxeo.runtime.test.runner.FeaturesRunner$RulesFactory$1.evaluate(FeaturesRunner.java:435) at org.nuxeo.runtime.test.runner.FeaturesRunner$BeforeClassStatement.evaluate(FeaturesRunner.java:234) at org.junit.rules.RunRules.evaluate(RunRules.java:20) at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306) at org.junit.runners.ParentRunner.run(ParentRunner.java:413) at org.junit.runners.Suite.runChild(Suite.java:128) at org.junit.runners.Suite.runChild(Suite.java:27) at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329) at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293) at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306) at org.junit.runners.ParentRunner.run(ParentRunner.java:413) at org.apache.maven.surefire.junitcore.JUnitCore.run(JUnitCore.java:55) at org.apache.maven.surefire.junitcore.JUnitCoreWrapper.createRequestAndRun(JUnitCoreWrapper.java:137) at org.apache.maven.surefire.junitcore.JUnitCoreWrapper.executeEager(JUnitCoreWrapper.java:107) at org.apache.maven.surefire.junitcore.JUnitCoreWrapper.execute(JUnitCoreWrapper.java:83) at org.apache.maven.surefire.junitcore.JUnitCoreWrapper.execute(JUnitCoreWrapper.java:75) at org.apache.maven.surefire.junitcore.JUnitCoreProvider.invoke(JUnitCoreProvider.java:158) at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384) at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345) at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126) at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418) Caused by: javax.transaction.RollbackException: Unable to commit: Transaction timeout at org.apache.geronimo.transaction.manager.TransactionImpl.commit(TransactionImpl.java:267) at org.apache.geronimo.transaction.manager.TransactionManagerImpl.commit(TransactionManagerImpl.java:252) at org.nuxeo.runtime.jtajca.NuxeoContainer$UserTransactionImpl.commit(NuxeoContainer.java:538) at org.nuxeo.runtime.transaction.TransactionHelper.commitOrRollbackTransaction(TransactionHelper.java:393) ... 60 more Sortie standard 2020-02-27 17:00:00,008 [Quartz_Worker-1] ERROR [EventServiceImpl] Exception during findRetentionExpired sync listener execution, continuing to run other listeners java.lang.NullPointerException: null at org.nuxeo.ecm.core.security.RetentionExpiredFinderListener.handleEvent(RetentionExpiredFinderListener.java:53) ~[classes/:?] at org.nuxeo.ecm.core.event.impl.EventServiceImpl.fireEvent(EventServiceImpl.java:241) [classes/:?] at org.nuxeo.ecm.core.scheduler.EventJob.execute(EventJob.java:107) [classes/:?] at org.nuxeo.ecm.core.scheduler.EventJob.execute(EventJob.java:65) [classes/:?] at org.quartz.core.JobRunShell.run(JobRunShell.java:202) [quartz-2.3.2.jar:?] at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573) [quartz-2.3.2.jar:?]
TestPageProviderCurrentPage.testPageProviderCurrentPageWithoutPageSize
Run unit tests / Run postgresql unit tests / org.nuxeo.ecm.platform.query.core.TestPageProviderCurrentPage.testPageProviderCurrentPageWithoutPageSize java.lang.AssertionError: invoke on features error [Holder [type=class org.nuxeo.ecm.core.test.CoreFeature], Holder [type=class org.nuxeo.ecm.core.bulk.CoreBulkFeature], Holder [type=class org.nuxeo.ecm.core.io.CoreIOFeature], Holder [type=class org.nuxeo.ecm.core.work.WorkManagerFeature], Holder [type=class org.nuxeo.ecm.core.api.local.DummyLoginFeature], Holder [type=class org.nuxeo.runtime.stream.RuntimeStreamFeature], Holder [type=class org.nuxeo.runtime.test.runner.TransactionalFeature], Holder [type=class org.nuxeo.runtime.cluster.ClusterFeature], Holder [type=class org.nuxeo.runtime.test.runner.RuntimeFeature], Holder [type=class org.nuxeo.runtime.test.runner.WithFrameworkPropertyFeature], Holder [type=class org.nuxeo.runtime.test.runner.RandomBug$Feature], Holder [type=class org.nuxeo.runtime.test.runner.ConditionalIgnoreRule$Feature], Holder [type=class org.nuxeo.runtime.test.runner.MDCFeature]] at org.nuxeo.runtime.test.runner.FeaturesLoader.apply(FeaturesLoader.java:114) at org.nuxeo.runtime.test.runner.FeaturesLoader.apply(FeaturesLoader.java:104) at org.nuxeo.runtime.test.runner.FeaturesRunner.afterTeardown(FeaturesRunner.java:205) at org.nuxeo.runtime.test.runner.FeaturesRunner$AfterTeardownStatement.evaluate(FeaturesRunner.java:364) at org.nuxeo.runtime.test.runner.FeaturesRunner$RulesFactory$1.evaluate(FeaturesRunner.java:435) at org.nuxeo.runtime.test.runner.RuntimeDeployment$DeploymentStatement.evaluate(RuntimeDeployment.java:326) at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306) at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63) at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329) at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293) at org.nuxeo.runtime.test.runner.FeaturesRunner$AfterClassStatement.evaluate(FeaturesRunner.java:251) at org.nuxeo.runtime.test.runner.FeaturesRunner$RulesFactory$1.evaluate(FeaturesRunner.java:435) at org.nuxeo.runtime.test.runner.FeaturesRunner$BeforeClassStatement.evaluate(FeaturesRunner.java:234) at org.junit.rules.RunRules.evaluate(RunRules.java:20) at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306) at org.junit.runners.ParentRunner.run(ParentRunner.java:413) at org.junit.runners.Suite.runChild(Suite.java:128) at org.junit.runners.Suite.runChild(Suite.java:27) at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329) at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293) at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306) at org.junit.runners.ParentRunner.run(ParentRunner.java:413) at org.apache.maven.surefire.junitcore.JUnitCore.run(JUnitCore.java:55) at org.apache.maven.surefire.junitcore.JUnitCoreWrapper.createRequestAndRun(JUnitCoreWrapper.java:137) at org.apache.maven.surefire.junitcore.JUnitCoreWrapper.executeEager(JUnitCoreWrapper.java:107) at org.apache.maven.surefire.junitcore.JUnitCoreWrapper.execute(JUnitCoreWrapper.java:83) at org.apache.maven.surefire.junitcore.JUnitCoreWrapper.execute(JUnitCoreWrapper.java:75) at org.apache.maven.surefire.junitcore.JUnitCoreProvider.invoke(JUnitCoreProvider.java:158) at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384) at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345) at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126) at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418) Suppressed: org.nuxeo.runtime.transaction.TransactionRuntimeException: Transaction has timed out at org.nuxeo.runtime.transaction.TransactionHelper.checkTransactionTimeout(TransactionHelper.java:237) at org.nuxeo.ecm.core.api.local.LocalSession.getSession(LocalSession.java:108) at org.nuxeo.ecm.core.api.AbstractSession.removeNotifyOneDoc(AbstractSession.java:1424) at org.nuxeo.ecm.core.api.AbstractSession.removeChildren(AbstractSession.java:1338) at org.nuxeo.ecm.core.test.CoreFeature.lambda$cleanupSession$1(CoreFeature.java:268) at org.nuxeo.runtime.transaction.TransactionHelper.lambda$runInNewTransaction$3(TransactionHelper.java:565) at org.nuxeo.runtime.transaction.TransactionHelper.runInTransaction(TransactionHelper.java:607) at org.nuxeo.runtime.transaction.TransactionHelper.lambda$runInNewTransaction$4(TransactionHelper.java:576) at org.nuxeo.runtime.transaction.TransactionHelper.runAndCleanupTransactionContext(TransactionHelper.java:661) at org.nuxeo.runtime.transaction.TransactionHelper.lambda$runWithoutTransaction$2(TransactionHelper.java:555) at org.nuxeo.runtime.transaction.TransactionHelper.lambda$runWithoutTransactionInternal$6(TransactionHelper.java:638) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) Suppressed: org.nuxeo.runtime.transaction.TransactionRuntimeException: Unable to commit: Transaction timeout at org.nuxeo.runtime.transaction.TransactionHelper.commitOrRollbackTransaction(TransactionHelper.java:410) at org.nuxeo.runtime.test.runner.TransactionalFeature.commitOrRollbackTransactionAfter(TransactionalFeature.java:156) at org.nuxeo.runtime.test.runner.TransactionalFeature.afterTeardown(TransactionalFeature.java:145) at org.nuxeo.runtime.test.runner.FeaturesRunner.lambda$afterTeardown$8(FeaturesRunner.java:205) at org.nuxeo.runtime.test.runner.FeaturesLoader.apply(FeaturesLoader.java:117) ... 40 more Caused by: javax.transaction.RollbackException: Unable to commit: Transaction timeout at org.apache.geronimo.transaction.manager.TransactionImpl.commit(TransactionImpl.java:267) at org.apache.geronimo.transaction.manager.TransactionManagerImpl.commit(TransactionManagerImpl.java:252) at org.nuxeo.runtime.jtajca.NuxeoContainer$UserTransactionImpl.commit(NuxeoContainer.java:538) at org.nuxeo.runtime.transaction.TransactionHelper.commitOrRollbackTransaction(TransactionHelper.java:393) ... 44 more
TestTagsMigrator.testMigrationThroughService
un unit tests / Run postgresql unit tests / org.nuxeo.ecm.platform.tag.TestTagsMigrator.testMigrationThroughService java.lang.AssertionError: invoke on features error [Holder [type=class org.nuxeo.ecm.core.test.CoreFeature], Holder [type=class org.nuxeo.ecm.core.bulk.CoreBulkFeature], Holder [type=class org.nuxeo.ecm.core.io.CoreIOFeature], Holder [type=class org.nuxeo.ecm.core.work.WorkManagerFeature], Holder [type=class org.nuxeo.ecm.core.api.local.DummyLoginFeature], Holder [type=class org.nuxeo.runtime.stream.RuntimeStreamFeature], Holder [type=class org.nuxeo.runtime.test.runner.TransactionalFeature], Holder [type=class org.nuxeo.runtime.cluster.ClusterFeature], Holder [type=class org.nuxeo.runtime.test.runner.RuntimeFeature], Holder [type=class org.nuxeo.runtime.test.runner.WithFrameworkPropertyFeature], Holder [type=class org.nuxeo.runtime.test.runner.RandomBug$Feature], Holder [type=class org.nuxeo.runtime.test.runner.ConditionalIgnoreRule$Feature], Holder [type=class org.nuxeo.runtime.test.runner.MDCFeature]] at org.nuxeo.runtime.test.runner.FeaturesLoader.apply(FeaturesLoader.java:114) at org.nuxeo.runtime.test.runner.FeaturesLoader.apply(FeaturesLoader.java:104) at org.nuxeo.runtime.test.runner.FeaturesRunner.afterTeardown(FeaturesRunner.java:205) at org.nuxeo.runtime.test.runner.FeaturesRunner$AfterTeardownStatement.evaluate(FeaturesRunner.java:364) at org.nuxeo.runtime.test.runner.FeaturesRunner$RulesFactory$1.evaluate(FeaturesRunner.java:435) at org.nuxeo.runtime.test.runner.RuntimeDeployment$DeploymentStatement.evaluate(RuntimeDeployment.java:326) at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306) at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63) at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329) at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293) at org.nuxeo.runtime.test.runner.FeaturesRunner$AfterClassStatement.evaluate(FeaturesRunner.java:251) at org.nuxeo.runtime.test.runner.FeaturesRunner$RulesFactory$1.evaluate(FeaturesRunner.java:435) at org.nuxeo.runtime.test.runner.FeaturesRunner$BeforeClassStatement.evaluate(FeaturesRunner.java:234) at org.junit.rules.RunRules.evaluate(RunRules.java:20) at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306) at org.junit.runners.ParentRunner.run(ParentRunner.java:413) at org.junit.runners.Suite.runChild(Suite.java:128) at org.junit.runners.Suite.runChild(Suite.java:27) at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329) at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293) at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306) at org.junit.runners.ParentRunner.run(ParentRunner.java:413) at org.apache.maven.surefire.junitcore.JUnitCore.run(JUnitCore.java:55) at org.apache.maven.surefire.junitcore.JUnitCoreWrapper.createRequestAndRun(JUnitCoreWrapper.java:137) at org.apache.maven.surefire.junitcore.JUnitCoreWrapper.executeEager(JUnitCoreWrapper.java:107) at org.apache.maven.surefire.junitcore.JUnitCoreWrapper.execute(JUnitCoreWrapper.java:83) at org.apache.maven.surefire.junitcore.JUnitCoreWrapper.execute(JUnitCoreWrapper.java:75) at org.apache.maven.surefire.junitcore.JUnitCoreProvider.invoke(JUnitCoreProvider.java:158) at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384) at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345) at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126) at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418) Suppressed: org.nuxeo.runtime.transaction.TransactionRuntimeException: Unable to commit: Transaction timeout at org.nuxeo.runtime.transaction.TransactionHelper.commitOrRollbackTransaction(TransactionHelper.java:410) at org.nuxeo.runtime.test.runner.TransactionalFeature.nextTransaction(TransactionalFeature.java:95) at org.nuxeo.runtime.test.runner.TransactionalFeature.nextTransaction(TransactionalFeature.java:78) at org.nuxeo.ecm.core.test.CoreFeature.waitForAsyncCompletion(CoreFeature.java:249) at org.nuxeo.ecm.core.test.CoreFeature.afterTeardown(CoreFeature.java:242) at org.nuxeo.runtime.test.runner.FeaturesRunner.lambda$afterTeardown$8(FeaturesRunner.java:205) at org.nuxeo.runtime.test.runner.FeaturesLoader.apply(FeaturesLoader.java:117) ... 40 more Caused by: javax.transaction.RollbackException: Unable to commit: Transaction timeout at org.apache.geronimo.transaction.manager.TransactionImpl.commit(TransactionImpl.java:267) at org.apache.geronimo.transaction.manager.TransactionManagerImpl.commit(TransactionManagerImpl.java:252) at org.nuxeo.runtime.jtajca.NuxeoContainer$UserTransactionImpl.commit(NuxeoContainer.java:538) at org.nuxeo.runtime.transaction.TransactionHelper.commitOrRollbackTransaction(TransactionHelper.java:393) ... 46 more
- is related to
-
NXP-28753 Fix TestTagsMigrator.testMigrationThroughService systematically failing in JX against PostgreSQL
-
- Resolved
-
-
-
- Resolved
-
-
-
- Resolved
-
-
NXP-28579 Run Nuxeo Platform unit tests against MongoDB and PostgreSQL
-
- Resolved
-
-
NXBT-3286 Run parallel multi environment unit tests in separate containers
-
- Resolved
-
-
-
- Resolved
-
(1 is related to)
