-
Type: Bug
-
Status: Resolved
-
Priority: Minor
-
Resolution: Fixed
-
Affects Version/s: None
-
Fix Version/s: coldstorage-2021.1, coldstorage-10.1, coldstorage-2023.0
-
Component/s: ColdStorage, Core, Glacier
-
Tags:
-
Team:PLATFORM
-
Sprint:nxplatform #83
Since NXP-31594, we are seeing random failure in ColdStorage utests https://jenkins.platform.dev.nuxeo.com/blue/organizations/jenkins/coldstorage%2Fnuxeo-coldstorage/detail/PR-240/1/tests/ :
Caused by: java.io.IOException: com.amazonaws.services.s3.model.AmazonS3Exception: The specified key does not exist. (Service: Amazon S3; Status Code: 404; Error Code: NoSuchKey; Request ID: JF47M55QD1WH45YQ; S3 Extended Request ID: DEY2UrJxlnch+nJI0X8IgLhmkLW+5fxdoxZB8I8cjAjlSrAvx9V8VT5xleqNsGcjSutXxAbdq+M=; Proxy: null), S3 Extended Request ID: DEY2UrJxlnch+nJI0X8IgLhmkLW+5fxdoxZB8I8cjAjlSrAvx9V8VT5xleqNsGcjSutXxAbdq+M= at org.nuxeo.ecm.blob.s3.S3BlobStore.writeBlobProperties(S3BlobStore.java:849) at org.nuxeo.ecm.core.blob.CachingBlobStore.writeBlobProperties(CachingBlobStore.java:238) at org.nuxeo.ecm.core.blob.BlobStoreBlobProvider.updateBlob(BlobStoreBlobProvider.java:249) at org.nuxeo.coldstorage.service.ColdStorageServiceImpl.proceedMoveToColdStorage(ColdStorageServiceImpl.java:277) ... 53 more Caused by: com.amazonaws.services.s3.model.AmazonS3Exception: The specified key does not exist. (Service: Amazon S3; Status Code: 404; Error Code: NoSuchKey; Request ID: JF47M55QD1WH45YQ; S3 Extended Request ID: DEY2UrJxlnch+nJI0X8IgLhmkLW+5fxdoxZB8I8cjAjlSrAvx9V8VT5xleqNsGcjSutXxAbdq+M=; Proxy: null), S3 Extended Request ID: DEY2UrJxlnch+nJI0X8IgLhmkLW+5fxdoxZB8I8cjAjlSrAvx9V8VT5xleqNsGcjSutXxAbdq+M= at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1879) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleServiceErrorResponse(AmazonHttpClient.java:1418) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1387) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1157) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:814) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:781) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:755) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:715) at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:697) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:561) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:541) at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5456) at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5403) at com.amazonaws.services.s3.AmazonS3Client.copyObject(AmazonS3Client.java:2078) at org.nuxeo.ecm.blob.s3.S3BlobStore.writeBlobProperties(S3BlobStore.java:832) ... 56 more
We suspect it is because all the tests are using the same blob content to create documents and we would need NXP-31714 to wait for the end of each stream processing. The blob domain events from the previous test execution are handled during the next test execution resulting in an unexpected deletion of the current test data.
Other random failure, a postgres concurrent update :
org.nuxeo.ecm.core.api.ConcurrentUpdateException: Failed to save session, Concurrent update at org.nuxeo.ecm.core.storage.sql.jdbc.JDBCConnection.checkConcurrentUpdate(JDBCConnection.java:172) at org.nuxeo.ecm.core.storage.sql.jdbc.JDBCRowMapper.insertSimpleRows(JDBCRowMapper.java:543) at org.nuxeo.ecm.core.storage.sql.jdbc.JDBCRowMapper.writeCreates(JDBCRowMapper.java:454) at org.nuxeo.ecm.core.storage.sql.jdbc.JDBCRowMapper.write(JDBCRowMapper.java:431) at org.nuxeo.ecm.core.storage.sql.SoftRefCachingRowMapper.write(SoftRefCachingRowMapper.java:366) at org.nuxeo.ecm.core.storage.sql.SessionImpl.doFlush(SessionImpl.java:277) at org.nuxeo.ecm.core.storage.sql.SessionImpl.flush(SessionImpl.java:249) at org.nuxeo.ecm.core.storage.sql.SessionImpl.save(SessionImpl.java:229) at org.nuxeo.ecm.core.storage.sql.coremodel.SQLSession.save(SQLSession.java:127) at org.nuxeo.ecm.core.api.AbstractSession.save(AbstractSession.java:1556) at org.nuxeo.ecm.automation.OperationContext.close(OperationContext.java:210) at org.nuxeo.coldstorage.operations.AbstractTestColdStorageOperation.moveContentToColdStorage(AbstractTestColdStorageOperation.java:96) at org.nuxeo.coldstorage.operations.MoveToColdStorageTest.shouldMoveDocsToColdStorage(MoveToColdStorageTest.java:124) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:568) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.nuxeo.runtime.test.runner.FeaturesRunner$BeforeMethodRunStatement.evaluate(FeaturesRunner.java:457) at org.nuxeo.runtime.test.runner.FeaturesRunner$BeforeSetupStatement.evaluate(FeaturesRunner.java:471) at org.nuxeo.runtime.test.runner.FeaturesRunner.evaluateRunnable(FeaturesRunner.java:328) at org.nuxeo.runtime.test.runner.FeaturesRunner$AfterMethodRunStatement.evaluate(FeaturesRunner.java:493) at org.nuxeo.runtime.test.runner.FeaturesRunner.evaluateRunnable(FeaturesRunner.java:328) at org.nuxeo.runtime.test.runner.FeaturesRunner$AfterTeardownStatement.evaluate(FeaturesRunner.java:506) at org.nuxeo.runtime.test.runner.FeaturesRunner$RulesFactory$1.evaluate(FeaturesRunner.java:576) at org.nuxeo.runtime.test.runner.RuntimeDeployment$DeploymentStatement.evaluate(RuntimeDeployment.java:293) at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306) at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63) at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329) at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293) at org.nuxeo.runtime.test.runner.FeaturesRunner$AfterClassStatement.evaluate(FeaturesRunner.java:404) at org.nuxeo.runtime.test.runner.FeaturesRunner$RulesFactory$1.evaluate(FeaturesRunner.java:576) at org.nuxeo.runtime.test.runner.FeaturesRunner$BeforeClassStatement.evaluate(FeaturesRunner.java:387) at org.junit.rules.RunRules.evaluate(RunRules.java:20) at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306) at org.junit.runners.ParentRunner.run(ParentRunner.java:413) at org.junit.runners.Suite.runChild(Suite.java:128) at org.junit.runners.Suite.runChild(Suite.java:27) at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329) at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293) at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306) at org.junit.runners.ParentRunner.run(ParentRunner.java:413) at org.apache.maven.surefire.junitcore.JUnitCore.run(JUnitCore.java:55) at org.apache.maven.surefire.junitcore.JUnitCoreWrapper.createRequestAndRun(JUnitCoreWrapper.java:137) at org.apache.maven.surefire.junitcore.JUnitCoreWrapper.executeEager(JUnitCoreWrapper.java:107) at org.apache.maven.surefire.junitcore.JUnitCoreWrapper.execute(JUnitCoreWrapper.java:83) at org.apache.maven.surefire.junitcore.JUnitCoreWrapper.execute(JUnitCoreWrapper.java:75) at org.apache.maven.surefire.junitcore.JUnitCoreProvider.invoke(JUnitCoreProvider.java:158) at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384) at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345) at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126) at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418) Suppressed: java.lang.AssertionError: Error while invoking afterTeardown on features: [org.nuxeo.coldstorage.DummyColdStorageFeature, org.nuxeo.coldstorage.ColdStorageFeature, org.nuxeo.runtime.test.runner.LogCaptureFeature, org.nuxeo.ecm.platform.picture.core.ImagingFeature, org.nuxeo.ecm.automation.core.AutomationCoreFeature, org.nuxeo.ecm.core.test.CoreFeature, org.nuxeo.ecm.core.bulk.CoreBulkFeature, org.nuxeo.ecm.core.work.WorkManagerFeature, org.nuxeo.ecm.core.event.CoreEventFeature, org.nuxeo.ecm.core.api.local.DummyLoginFeature, org.nuxeo.runtime.stream.RuntimeStreamFeature, org.nuxeo.runtime.test.runner.TransactionalFeature, org.nuxeo.runtime.cluster.ClusterFeature, org.nuxeo.runtime.test.runner.RuntimeFeature, org.nuxeo.runtime.test.runner.WithFrameworkPropertyFeature, org.nuxeo.runtime.test.runner.RandomBug$Feature, org.nuxeo.runtime.test.runner.ConditionalIgnoreRule$Feature, org.nuxeo.runtime.test.runner.MDCFeature] at org.nuxeo.runtime.test.runner.FeaturesRunner.apply(FeaturesRunner.java:314) at org.nuxeo.runtime.test.runner.FeaturesRunner.apply(FeaturesRunner.java:286) at org.nuxeo.runtime.test.runner.FeaturesRunner.afterTeardown(FeaturesRunner.java:279) at org.nuxeo.runtime.test.runner.FeaturesRunner$AfterTeardownStatement.lambda$evaluate$0(FeaturesRunner.java:506) at org.nuxeo.runtime.test.runner.FeaturesRunner.evaluateRunnable(FeaturesRunner.java:338) ... 38 more Suppressed: org.nuxeo.runtime.transaction.TransactionRuntimeException: Unable to commit: transaction marked for rollback at org.nuxeo.runtime.transaction.TransactionHelper.commitOrRollbackTransaction(TransactionHelper.java:443) at org.nuxeo.runtime.test.runner.TransactionalFeature.nextTransaction(TransactionalFeature.java:95) at org.nuxeo.runtime.test.runner.TransactionalFeature.nextTransaction(TransactionalFeature.java:78) at org.nuxeo.ecm.core.test.CoreFeature.waitForAsyncCompletion(CoreFeature.java:231) at org.nuxeo.ecm.core.test.CoreFeature.afterTeardown(CoreFeature.java:224) at org.nuxeo.runtime.test.runner.FeaturesRunner.lambda$afterTeardown$9(FeaturesRunner.java:279) at org.nuxeo.runtime.test.runner.FeaturesRunner.apply(FeaturesRunner.java:300) ... 42 more Caused by: javax.transaction.RollbackException: Unable to commit: transaction marked for rollback at org.apache.geronimo.transaction.manager.TransactionImpl.commit(TransactionImpl.java:294) at org.apache.geronimo.transaction.manager.TransactionManagerImpl.commit(TransactionManagerImpl.java:252) at org.nuxeo.runtime.jtajca.NuxeoContainer$UserTransactionImpl.commit(NuxeoContainer.java:418) at org.nuxeo.runtime.transaction.TransactionHelper.commitOrRollbackTransaction(TransactionHelper.java:428) ... 48 more Caused by: org.apache.geronimo.transaction.manager.SetRollbackOnlyException: setRollbackOnly() called. See stacktrace for origin at org.apache.geronimo.transaction.manager.TransactionImpl.setRollbackOnly(TransactionImpl.java:147) at org.apache.geronimo.transaction.manager.TransactionManagerImpl.setRollbackOnly(TransactionManagerImpl.java:243) at org.nuxeo.runtime.jtajca.NuxeoContainer$UserTransactionImpl.setRollbackOnly(NuxeoContainer.java:358) at org.nuxeo.runtime.transaction.TransactionHelper.setTransactionRollbackOnly(TransactionHelper.java:531) at org.nuxeo.ecm.core.storage.sql.SessionImpl.end(SessionImpl.java:1262) at org.nuxeo.ecm.core.storage.sql.coremodel.SQLSession.end(SQLSession.java:867) at org.nuxeo.ecm.core.repository.RepositoryService$SessionSynchronization.beforeCompletion(RepositoryService.java:344) at org.apache.geronimo.transaction.manager.TransactionImpl.beforeCompletion(TransactionImpl.java:549) at org.apache.geronimo.transaction.manager.TransactionImpl.beforeCompletion(TransactionImpl.java:533) at org.apache.geronimo.transaction.manager.TransactionImpl.beforePrepare(TransactionImpl.java:435) at org.apache.geronimo.transaction.manager.TransactionImpl.commit(TransactionImpl.java:283) ... 51 more Caused by: org.nuxeo.ecm.core.api.ConcurrentUpdateException: Concurrent update at org.nuxeo.ecm.core.storage.sql.jdbc.JDBCConnection.checkConcurrentUpdate(JDBCConnection.java:172) at org.nuxeo.ecm.core.storage.sql.jdbc.JDBCMapper.updateReadAcls(JDBCMapper.java:898) at org.nuxeo.ecm.core.storage.sql.SoftRefCachingMapper.updateReadAcls(SoftRefCachingMapper.java:122) at org.nuxeo.ecm.core.storage.sql.SessionImpl.updateReadAcls(SessionImpl.java:1186) at org.nuxeo.ecm.core.storage.sql.SessionImpl.flush(SessionImpl.java:251) at org.nuxeo.ecm.core.storage.sql.SessionImpl.end(SessionImpl.java:1260) ... 57 more Caused by: org.postgresql.util.PSQLException: ERROR: current transaction is aborted, commands ignored until end of transaction block at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2676) at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2366) at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:356) at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:496) at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:413) at org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:333) at org.postgresql.jdbc.PgStatement.executeCachedSql(PgStatement.java:319) at org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:295) at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:290) at org.apache.commons.dbcp2.DelegatingStatement.execute(DelegatingStatement.java:193) at org.apache.commons.dbcp2.DelegatingStatement.execute(DelegatingStatement.java:193) at org.nuxeo.ecm.core.storage.sql.jdbc.JDBCMapper.updateReadAcls(JDBCMapper.java:895) ... 61 more Caused by: org.postgresql.util.PSQLException: ERROR: duplicate key value violates unique constraint "hierarchy_unique_child" Detail: Key (parentid, name)=(be2284d8-ebeb-4778-a500-f500e34e802b, MyFile.1678378766894) already exists. at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2676) at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2366) at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:560) at org.postgresql.jdbc.PgStatement.internalExecuteBatch(PgStatement.java:893) at org.postgresql.jdbc.PgStatement.executeBatch(PgStatement.java:916) at org.postgresql.jdbc.PgPreparedStatement.executeBatch(PgPreparedStatement.java:1684) at org.apache.commons.dbcp2.DelegatingStatement.executeBatch(DelegatingStatement.java:241) at org.apache.commons.dbcp2.DelegatingStatement.executeBatch(DelegatingStatement.java:241) at org.nuxeo.ecm.core.storage.sql.jdbc.JDBCRowMapper.insertSimpleRows(JDBCRowMapper.java:527) at org.nuxeo.ecm.core.storage.sql.jdbc.JDBCRowMapper.writeCreates(JDBCRowMapper.java:454) at org.nuxeo.ecm.core.storage.sql.jdbc.JDBCRowMapper.write(JDBCRowMapper.java:431) at org.nuxeo.ecm.core.storage.sql.SoftRefCachingRowMapper.write(SoftRefCachingRowMapper.java:366) at org.nuxeo.ecm.core.storage.sql.SessionImpl.doFlush(SessionImpl.java:277) at org.nuxeo.ecm.core.storage.sql.SessionImpl.flush(SessionImpl.java:249) at org.nuxeo.ecm.core.storage.sql.SessionImpl.save(SessionImpl.java:229) at org.nuxeo.ecm.core.storage.sql.coremodel.SQLSession.save(SQLSession.java:127) at org.nuxeo.ecm.core.api.AbstractSession.save(AbstractSession.java:1556) at org.nuxeo.ecm.automation.OperationContext.close(OperationContext.java:210) at org.nuxeo.coldstorage.operations.AbstractTestColdStorageOperation.moveContentToColdStorage(AbstractTestColdStorageOperation.java:96) at org.nuxeo.coldstorage.operations.MoveToColdStorageTest.shouldMoveDocsToColdStorage(MoveToColdStorageTest.java:124) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:568) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.nuxeo.runtime.test.runner.FeaturesRunner$BeforeMethodRunStatement.evaluate(FeaturesRunner.java:457) at org.nuxeo.runtime.test.runner.FeaturesRunner$BeforeSetupStatement.evaluate(FeaturesRunner.java:471) at org.nuxeo.runtime.test.runner.FeaturesRunner.evaluateRunnable(FeaturesRunner.java:328) at org.nuxeo.runtime.test.runner.FeaturesRunner$AfterMethodRunStatement.evaluate(FeaturesRunner.java:493) at org.nuxeo.runtime.test.runner.FeaturesRunner.evaluateRunnable(FeaturesRunner.java:328) ... 38 more Caused by: java.sql.BatchUpdateException: Batch entry 6 INSERT INTO "hierarchy" ("id", "parentid", "pos", "name", "isproperty", "primarytype", "mixintypes", "ischeckedin", "baseversionid", "majorversion", "minorversion", "isversion", "isrecord", "retainedproperties", "retainuntil", "haslegalhold", "isretentionactive", "istrashed", "systemchangetoken", "changetoken") VALUES ('b53e38a1-48c8-4eff-abcf-98edd6072aa2', 'be2284d8-ebeb-4778-a500-f500e34e802b', NULL, 'MyFile.1678378766894', 'FALSE', 'File', ?, NULL, NULL, 0, 0, NULL, NULL, NULL, NULL, NULL, NULL, NULL, 0, 0) was aborted: ERROR: duplicate key value violates unique constraint "hierarchy_unique_child" Detail: Key (parentid, name)=(be2284d8-ebeb-4778-a500-f500e34e802b, MyFile.1678378766894) already exists. Call getNextException to see other errors in the batch. at org.postgresql.jdbc.BatchResultHandler.handleError(BatchResultHandler.java:165) at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2367) at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:560) at org.postgresql.jdbc.PgStatement.internalExecuteBatch(PgStatement.java:893) at org.postgresql.jdbc.PgStatement.executeBatch(PgStatement.java:916) at org.postgresql.jdbc.PgPreparedStatement.executeBatch(PgPreparedStatement.java:1684) at org.apache.commons.dbcp2.DelegatingStatement.executeBatch(DelegatingStatement.java:241) at org.apache.commons.dbcp2.DelegatingStatement.executeBatch(DelegatingStatement.java:241) at org.nuxeo.ecm.core.storage.sql.jdbc.JDBCRowMapper.insertSimpleRows(JDBCRowMapper.java:527) ... 62 more Caused by: [CIRCULAR REFERENCE: org.postgresql.util.PSQLException: ERROR: duplicate key value violates unique constraint "hierarchy_unique_child" Detail: Key (parentid, name)=(be2284d8-ebeb-4778-a500-f500e34e802b, MyFile.1678378766894) already exists.]