Uploaded image for project: 'Nuxeo Platform'
  1. Nuxeo Platform
  2. NXP-24342

Fix startup of FunkLoad credential server

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Won't Fix
    • Affects Version/s: 9.10
    • Fix Version/s: None
    • Component/s: Tests

      Description

      Lots of FunkLoad "No server reachable" since January the 22sd.

      Seen:
      https://qa.nuxeo.org/jenkins/job/Deploy/job/IT-nuxeo-master-tests-cap-tomcat-mongodb/625/
      https://qa.nuxeo.org/jenkins/job/Deploy/job/IT-nuxeo-master-tests-cap-tomcat-mysql/605/
      https://qa.nuxeo.org/jenkins/job/Deploy/job/IT-nuxeo-master-tests-cap-tomcat-postgresql/622/
      This is at least 3/4 of all the failures for these IT jobs

      Typical output:

      05:38:04 
05:38:04 funkload-tests.run-funkload-tests:
05:38:04 [INFO]     [mkdir] Created dir: /opt/jenkins/workspace/Deploy/IT-nuxeo-master-tests-cap-tomcat-mongodb/nuxeo/nuxeo-distribution/nuxeo-jsf-ui-funkload-tests/target/results
05:38:04 [INFO]     [mkdir] Created dir: /opt/jenkins/workspace/Deploy/IT-nuxeo-master-tests-cap-tomcat-mongodb/nuxeo/nuxeo-distribution/nuxeo-jsf-ui-funkload-tests/target/log
05:38:04 [INFO] Managed version set on org.nuxeo:nuxeo-ftest:zip:1.8.4-SNAPSHOT
05:38:04 [INFO]     [unzip] Expanding: /opt/jenkins/.m2/repository/org/nuxeo/nuxeo-ftest/1.8.4-SNAPSHOT/nuxeo-ftest-1.8.4-SNAPSHOT.zip into /opt/jenkins/workspace/Deploy/IT-nuxeo-master-tests-cap-tomcat-mongodb/nuxeo/nuxeo-distribution/nuxeo-jsf-ui-funkload-tests/target
05:38:05 
05:38:05 funkload-tests.funkload-ctl-daemons:
05:38:05 [INFO]      [exec] Starting monitor server at http://0.0.0.0:55502/ as daemon.
05:38:05 [INFO]      [exec] Starting file_credential server at http://localhost:55501/ as daemon.
05:38:06 [INFO] Managed version set on org.nuxeo:nuxeo-ftest:zip:1.8.4-SNAPSHOT
05:38:06 [INFO]     [unzip] Expanding: /opt/jenkins/.m2/repository/org/nuxeo/nuxeo-ftest/1.8.4-SNAPSHOT/nuxeo-ftest-1.8.4-SNAPSHOT.zip into /opt/jenkins/workspace/Deploy/IT-nuxeo-master-tests-cap-tomcat-mongodb/nuxeo/nuxeo-distribution/nuxeo-jsf-ui-funkload-tests/target
05:38:06 
05:38:06 funkload-tests.funkload-ctl-daemons:
05:38:06 [INFO]      [exec] No server reachable at http://0.0.0.0:55502/.
05:38:06 [ERROR]      [exec] Result: 255
05:38:06 [INFO]      [exec] No server reachable at http://localhost:55501/.
05:38:06 [ERROR] 
05:38:06 BUILD FAILED - at 1/31/18 5:38 AM
05:38:06 /opt/jenkins/workspace/Deploy/IT-nuxeo-master-tests-cap-tomcat-mongodb/nuxeo/nuxeo-distribution/nuxeo-jsf-ui-funkload-tests/target/nuxeo-ftest.xml:627: The following error occurred while executing this line:
05:38:06 /opt/jenkins/workspace/Deploy/IT-nuxeo-master-tests-cap-tomcat-mongodb/nuxeo/nuxeo-distribution/nuxeo-jsf-ui-funkload-tests/target/nuxeo-ftest.xml:572: exec returned: 255
05:38:06 
05:38:06 Total time: 2 seconds
05:38:06 [ERROR] The following error occurred while executing this line:
05:38:06 /opt/jenkins/workspace/Deploy/IT-nuxeo-master-tests-cap-tomcat-mongodb/nuxeo/nuxeo-distribution/nuxeo-jsf-ui-funkload-tests/target/nuxeo-ftest.xml:572: exec returned: 255
05:38:06 /opt/jenkins/workspace/Deploy/IT-nuxeo-master-tests-cap-tomcat-mongodb/nuxeo/nuxeo-distribution/nuxeo-jsf-ui-funkload-tests/target/nuxeo-ftest.xml:627: The following error occurred while executing this line:
05:38:06 /opt/jenkins/workspace/Deploy/IT-nuxeo-master-tests-cap-tomcat-mongodb/nuxeo/nuxeo-distribution/nuxeo-jsf-ui-funkload-tests/target/nuxeo-ftest.xml:572: exec returned: 255
05:38:06 	at org.apache.tools.ant.ProjectHelper.addLocationToBuildException(ProjectHelper.java:568)
05:38:06 	at org.apache.tools.ant.taskdefs.Ant.execute(Ant.java:443)
05:38:06 	at org.apache.tools.ant.taskdefs.CallTarget.execute(CallTarget.java:105)
05:38:06 	at org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:293)
05:38:06 	at sun.reflect.GeneratedMethodAccessor27.invoke(Unknown Source)
05:38:06 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
05:38:06 	at java.lang.reflect.Method.invoke(Method.java:498)
05:38:06 	at org.apache.tools.ant.dispatch.DispatchUtils.execute(DispatchUtils.java:106)
05:38:06 	at org.apache.tools.ant.Task.perform(Task.java:348)
05:38:06 	at org.apache.tools.ant.Target.execute(Target.java:435)
05:38:06 	at org.apache.tools.ant.Target.performTasks(Target.java:456)
05:38:06 	at org.apache.tools.ant.Project.executeSortedTargets(Project.java:1405)
05:38:06 	at org.apache.tools.ant.Project.executeTarget(Project.java:1376)
05:38:06 	at org.apache.tools.ant.helper.DefaultExecutor.executeTargets(DefaultExecutor.java:41)
05:38:06 	at org.nuxeo.build.ant.AntClient.run(AntClient.java:159)
05:38:06 	at org.nuxeo.build.maven.AntBuildMojo.execute(AntBuildMojo.java:254)
05:38:06 	at org.nuxeo.build.maven.IntegrationTestMojo.execute(IntegrationTestMojo.java:103)
05:38:06 	at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:134)
05:38:06 	at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:207)
05:38:06 	at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
05:38:06 	at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
05:38:06 	at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116)
05:38:06 	at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:80)
05:38:06 	at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
05:38:06 	at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128)
05:38:06 	at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:307)
05:38:06 	at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:193)
05:38:06 	at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:106)
05:38:06 	at org.apache.maven.cli.MavenCli.execute(MavenCli.java:863)
05:38:06 	at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:288)
05:38:06 	at org.apache.maven.cli.MavenCli.main(MavenCli.java:199)
05:38:06 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
05:38:06 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
05:38:06 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
05:38:06 	at java.lang.reflect.Method.invoke(Method.java:498)
05:38:06 	at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289)
05:38:06 	at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229)
05:38:06 	at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)
05:38:06 	at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)
05:38:06 Caused by: /opt/jenkins/workspace/Deploy/IT-nuxeo-master-tests-cap-tomcat-mongodb/nuxeo/nuxeo-distribution/nuxeo-jsf-ui-funkload-tests/target/nuxeo-ftest.xml:572: exec returned: 255
05:38:06 	at org.apache.tools.ant.taskdefs.ExecTask.runExecute(ExecTask.java:644)
05:38:06 	at org.apache.tools.ant.taskdefs.ExecTask.runExec(ExecTask.java:670)
05:38:06 	at org.apache.tools.ant.taskdefs.ExecTask.execute(ExecTask.java:496)
05:38:06 	at org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:293)
05:38:06 	at sun.reflect.GeneratedMethodAccessor27.invoke(Unknown Source)
05:38:06 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
05:38:06 	at java.lang.reflect.Method.invoke(Method.java:498)
05:38:06 	at org.apache.tools.ant.dispatch.DispatchUtils.execute(DispatchUtils.java:106)
05:38:06 	at org.apache.tools.ant.Task.perform(Task.java:348)
05:38:06 	at org.apache.tools.ant.Target.execute(Target.java:435)
05:38:06 	at org.apache.tools.ant.Target.performTasks(Target.java:456)
05:38:06 	at org.apache.tools.ant.Project.executeSortedTargets(Project.java:1405)
05:38:06 	at org.apache.tools.ant.helper.SingleCheckExecutor.executeTargets(SingleCheckExecutor.java:38)
05:38:06 	at org.apache.tools.ant.Project.executeTargets(Project.java:1260)
05:38:06 	at org.apache.tools.ant.taskdefs.Ant.execute(Ant.java:441)
05:38:06 	... 37 more
05:38:06 [INFO] 
05:38:06 [INFO] --- ant-assembly-maven-plugin:2.1.0:build (stop-nuxeo) @ nuxeo-jsf-ui-funkload-tests ---
05:38:06 [INFO] Active Maven profiles:
05:38:06 qa (source: external)
05:38:06 qa (source: org.nuxeo.ecm.distribution:nuxeo-distribution:10.1-I20180131_0125)
05:38:06 nightly (source: org.nuxeo.ecm.distribution:nuxeo-distribution:10.1-I20180131_0125)
05:38:06 nightly (source: org.nuxeo:nuxeo-ecm:10.1-I20180131_0125)
05:38:06 qa (source: org.nuxeo:nuxeo-ecm:10.1-I20180131_0125)
05:38:06 os-notwindows (source: org.nuxeo:nuxeo-ecm:10.1-I20180131_0125)
05:38:06 mongodb (source: org.nuxeo:nuxeo-ecm:10.1-I20180131_0125)
05:38:06 [INFO] Managed version set on org.nuxeo:nuxeo-ftest:zip:1.8.4-SNAPSHOT
05:38:06 [INFO]     [unzip] Expanding: /opt/jenkins/.m2/repository/org/nuxeo/nuxeo-ftest/1.8.4-SNAPSHOT/nuxeo-ftest-1.8.4-SNAPSHOT.zip into /opt/jenkins/workspace/Deploy/IT-nuxeo-master-tests-cap-tomcat-mongodb/nuxeo/nuxeo-distribution/nuxeo-jsf-ui-funkload-tests/target
05:38:07 
05:38:07 funkload-tests.stop-unix:
05:38:07 [INFO]      [exec] Detected Tomcat server.
05:38:07 [INFO]      [exec] Nuxeo home:          /opt/jenkins/workspace/Deploy/IT-nuxeo-master-tests-cap-tomcat-mongodb/tomcat-funkload
05:38:07 [INFO]      [exec] Nuxeo configuration: /opt/jenkins/workspace/Deploy/IT-nuxeo-master-tests-cap-tomcat-mongodb/tomcat-funkload/bin/nuxeo.conf
05:38:07 [INFO]      [exec] Include template: /opt/jenkins/workspace/Deploy/IT-nuxeo-master-tests-cap-tomcat-mongodb/tomcat-funkload/templates/common-base
05:38:07 [INFO]      [exec] Include template: /opt/jenkins/workspace/Deploy/IT-nuxeo-master-tests-cap-tomcat-mongodb/tomcat-funkload/templates/common
05:38:07 [INFO]      [exec] Include template: /opt/jenkins/workspace/Deploy/IT-nuxeo-master-tests-cap-tomcat-mongodb/tomcat-funkload/templates/default
05:38:07 [INFO]      [exec] Include template: /opt/jenkins/workspace/Deploy/IT-nuxeo-master-tests-cap-tomcat-mongodb/tomcat-funkload/templates/mongodb
05:38:10 [INFO]      [exec] Stopping server.....Server stopped.
05:38:10 
05:38:10 funkload-tests.stop:
05:38:10 [INFO] Managed version set on org.nuxeo:nuxeo-ftest:zip:1.8.4-SNAPSHOT
05:38:10 [INFO]     [unzip] Expanding: /opt/jenkins/.m2/repository/org/nuxeo/nuxeo-ftest/1.8.4-SNAPSHOT/nuxeo-ftest-1.8.4-SNAPSHOT.zip into /opt/jenkins/workspace/Deploy/IT-nuxeo-master-tests-cap-tomcat-mongodb/nuxeo/nuxeo-distribution/nuxeo-jsf-ui-funkload-tests/target
05:38:11 [INFO] 
05:38:11 [INFO] --- ant-assembly-maven-plugin:2.1.0:verify (run-tests) @ nuxeo-jsf-ui-funkload-tests ---
05:38:11 [INFO] ------------------------------------------------------------------------
05:38:11 [INFO] BUILD FAILURE
05:38:11 [INFO] ------------------------------------------------------------------------

      This happen every day between 5 and 6am this is the max activity with 19 dockers running, all IT db tests are running at the same time (9 jobs) they are trigger by https://qa.nuxeo.org/jenkins/job/Deploy/job/IT-nuxeo-master-tests-cap-tomcat/ the monitoring show also 7.10 and 8.10 multidb jobs

      All the funkload tests are almost concurrently run around 5:35, and may be there is an isolation pb on the funkload credentials port, that being said I don't see change in load since the 22th.

        Attachments

          Issue Links

          is related to

          Clean up - NXP-24344 Remove the FunkLoad tests

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

            Activity

              People

              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: