Uploaded image for project: 'Nuxeo Platform'
  1. Nuxeo Platform
  2. NXP-22712

The same test executed in distinct contexts must produce distinct result files

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Open
    • Priority: Critical
    • Resolution: Unresolved
    • Affects Version/s: 9.3-SNAPSHOT
    • Fix Version/s: QualifiedToSchedule
    • Component/s: Tests

      Description

      Raised by NXBT-1788
      The OAuth2ChallengeFixture.tokenShouldValidatePKCE seems to have been run twice (while not being annotated as RandomBug) and it generated an inconsistent state with a failed build without an archived failed test.

      First run with failure
      Running org.nuxeo.ecm.platform.oauth.tests.OAuth2ChallengeFixture
      btpool0-9 01:51:21,916 ERROR [AuthorizationRequest] {{fState,beforeRun}} No redirect URI set for OAuth2 client OAuth2Client(name=No redirect URI, id=no-redirect-uri, redirectURIs=[], enabled=true), at least one is required. Please make sure you update this OAuth2 client.
      btpool0-9 01:51:21,921 ERROR [AuthorizationRequest] {{fState,beforeRun}} The redirect URI http://redirect.uri set for OAuth2 client OAuth2Client(name=Not HTTPS, id=not-https, redirectURIs=[http://redirect.uri], enabled=true) is invalid: it must not be empty and start with https for security reasons. Please make sure you update this OAuth2 client.
      btpool0-9 01:51:21,923 ERROR [AuthorizationRequest] {{fState,beforeRun}} The redirect URI http://localhost.somecompany.com set for OAuth2 client OAuth2Client(name=Localhost as domain name, id=localhost-domain-name, redirectURIs=[http://localhost.somecompany.com], enabled=true) is invalid: it must not be empty and start with https for security reasons. Please make sure you update this OAuth2 client.
      btpool0-9 01:51:21,929 ERROR [AuthorizationRequest] {{fState,beforeRun}} The redirect URI http://redirect.uri set for OAuth2 client OAuth2Client(name=Not HTTPS, id=not-https, redirectURIs=[http://redirect.uri], enabled=true) is invalid: it must not be empty and start with https for security reasons. Please make sure you update this OAuth2 client.
      main 01:51:23,739 WARN  [CacheServiceImpl] {{fState,afterRun}{fSuite,class org.nuxeo.ecm.platform.oauth.tests.OAuth2ChallengeFixture}} Unregistery leaked contribution default-test-cache
      Tests run: 8, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 2.88 sec <<< FAILURE! - in org.nuxeo.ecm.platform.oauth.tests.OAuth2ChallengeFixture
      tokenShouldValidatePKCE(org.nuxeo.ecm.platform.oauth.tests.OAuth2ChallengeFixture)  Time elapsed: 0.215 sec  <<< FAILURE!
      java.lang.AssertionError: expected:<302> but was:<400>
      	at org.junit.Assert.fail(Assert.java:88)
      	at org.junit.Assert.failNotEquals(Assert.java:743)
      	at org.junit.Assert.assertEquals(Assert.java:118)
      	at org.junit.Assert.assertEquals(Assert.java:555)
      	at org.junit.Assert.assertEquals(Assert.java:542)
      	at org.nuxeo.ecm.platform.oauth.tests.OAuth2ChallengeFixture.getAuthorizationCode(OAuth2ChallengeFixture.java:422)
      	at org.nuxeo.ecm.platform.oauth.tests.OAuth2ChallengeFixture.getTokenResponse(OAuth2ChallengeFixture.java:447)
      	at org.nuxeo.ecm.platform.oauth.tests.OAuth2ChallengeFixture.tokenShouldValidatePKCE(OAuth2ChallengeFixture.java:318)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:498)
      	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
      	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
      	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
      	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
      	at org.nuxeo.runtime.test.runner.FeaturesRunner$BeforeMethodRunStatement.evaluate(FeaturesRunner.java:285)
      	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
      	at org.nuxeo.runtime.test.runner.FeaturesRunner$BeforeSetupStatement.evaluate(FeaturesRunner.java:301)
      	at org.nuxeo.runtime.test.runner.FeaturesRunner$AfterMethodRunStatement.evaluate(FeaturesRunner.java:332)
      	at org.nuxeo.runtime.test.runner.FeaturesRunner$AfterTeardownStatement.evaluate(FeaturesRunner.java:351)
      	at org.nuxeo.runtime.test.runner.FeaturesRunner$RulesFactory$1.evaluate(FeaturesRunner.java:429)
      	at org.nuxeo.runtime.test.runner.RuntimeDeployment$DeploymentStatement.evaluate(RuntimeDeployment.java:299)
      	at org.nuxeo.runtime.test.runner.RuntimeFeature$2$1.evaluate(RuntimeFeature.java:137)
      	at org.nuxeo.runtime.test.runner.FeaturesRunner$RulesFactory$1.evaluate(FeaturesRunner.java:429)
      	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
      	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
      	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
      	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
      	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
      	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
      	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
      	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
      	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
      	at org.nuxeo.runtime.test.runner.FeaturesRunner$AfterClassStatement.evaluate(FeaturesRunner.java:235)
      	at org.nuxeo.runtime.test.runner.FeaturesRunner$RulesFactory$1.evaluate(FeaturesRunner.java:429)
      	at org.nuxeo.runtime.test.runner.FeaturesRunner$BeforeClassStatement.evaluate(FeaturesRunner.java:218)
      	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
      	at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
      	at org.junit.runners.Suite.runChild(Suite.java:127)
      	at org.junit.runners.Suite.runChild(Suite.java:26)
      	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
      	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
      	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
      	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
      	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
      	at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
      	at org.junit.runners.Suite.runChild(Suite.java:127)
      	at org.junit.runners.Suite.runChild(Suite.java:26)
      	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
      	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
      	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
      	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
      	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
      	at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
      	at org.junit.runner.JUnitCore.run(JUnitCore.java:160)
      	at org.junit.runner.JUnitCore.run(JUnitCore.java:138)
      	at org.apache.maven.surefire.junitcore.JUnitCoreWrapper.createRequestAndRun(JUnitCoreWrapper.java:113)
      	at org.apache.maven.surefire.junitcore.JUnitCoreWrapper.executeEager(JUnitCoreWrapper.java:85)
      	at org.apache.maven.surefire.junitcore.JUnitCoreWrapper.execute(JUnitCoreWrapper.java:54)
      	at org.apache.maven.surefire.junitcore.JUnitCoreProvider.invoke(JUnitCoreProvider.java:134)
      	at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:200)
      	at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:153)
      	at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)
      
      Second run without failure
      Running org.nuxeo.ecm.platform.oauth.tests.OAuth2ChallengeFixture
      btpool1-8 01:51:24,805 ERROR [AuthorizationRequest] {{fState,beforeRun}} No redirect URI set for OAuth2 client OAuth2Client(name=No redirect URI, id=no-redirect-uri, redirectURIs=[], enabled=true), at least one is required. Please make sure you update this OAuth2 client.
      btpool1-8 01:51:24,808 ERROR [AuthorizationRequest] {{fState,beforeRun}} The redirect URI http://redirect.uri set for OAuth2 client OAuth2Client(name=Not HTTPS, id=not-https, redirectURIs=[http://redirect.uri], enabled=true) is invalid: it must not be empty and start with https for security reasons. Please make sure you update this OAuth2 client.
      btpool1-8 01:51:24,812 ERROR [AuthorizationRequest] {{fState,beforeRun}} The redirect URI http://localhost.somecompany.com set for OAuth2 client OAuth2Client(name=Localhost as domain name, id=localhost-domain-name, redirectURIs=[http://localhost.somecompany.com], enabled=true) is invalid: it must not be empty and start with https for security reasons. Please make sure you update this OAuth2 client.
      btpool1-8 01:51:24,818 ERROR [AuthorizationRequest] {{fState,beforeRun}} The redirect URI http://redirect.uri set for OAuth2 client OAuth2Client(name=Not HTTPS, id=not-https, redirectURIs=[http://redirect.uri], enabled=true) is invalid: it must not be empty and start with https for security reasons. Please make sure you update this OAuth2 client.
      StorageConfiguration: Deploying JDBC using DatabaseH2
      StorageConfiguration: Deploying a VCS repository
      Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.259 sec - in org.nuxeo.ecm.platform.oauth.tests.OAuth2ChallengeFixture

      The Maven build is failed but the JUnit result files are successful (the second run produced a result file which overwritten the first one):

      Final result
      Results :
      
      Failed tests: 
        OAuth2ChallengeFixture.tokenShouldValidatePKCE:318->getTokenResponse:447->getAuthorizationCode:422 expected:<302> but was:<400>
      
      Tests run: 28, Failures: 1, Errors: 0, Skipped: 0
      

      At the end, we get an inconsistent status:

      Should the build have succeed? Else, shouldn't the failed tests results have been erased? Why was it run twice?
      What is the effective status on those tests?

        Attachments

          Issue Links

            Activity

              People

              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated: