Summary

Nuxeo Studio Integration Tests (Studio ITests) are integration tests launched against all the supported Nuxeo platforms from 5.5 to 10.10. This proposal aims to change the way we host the source code of Nuxeo Studio integration, to make it easier to launch and in a more transparent way. More precisely, move all branches from repository nuxeo/nuxeo-studio-itests to nuxeo/nos.

Motivation

Why are we doing this?

• Make the "hidden" branches visible. Currently, the source code of each target platform is hosted in a separated branch. It is difficult to have the overview of all the existing tests.
• Improve search-ability. Having all tests inside Git workspace helps to search quickly, without branch switching.
• Facilitate test logic comparison between TPs.
• Run ITests without deploying NOS artifacts. Currently, artifacts generated by NOS must be deployed to Nexus, before being tested by Studio ITests. Such mechanism makes regressions happen and requires extra focus about the ITests results. Merging Studio ITests into NOS repository isolates the artifacts testing in a single Jenkins build. In other words, it is now possible to run tests without deployment.
• Preparation for Jenkins pipeline.
• Parallelism control. DevTools complained about the number of dockers we used when we launch all the ITests in parallel.

Non Goals

The following points are non-goals. We are NOT going to do them. They are written here to be more explicit:

• Make dependency shortcut. No, we are not going to merge integration tests with existing NOS Maven modules. They won't have the same Maven parent POM. We are aware of the dependency management risk it can bring. Each platform will have their own parent POM. No share between NOS and ITests, and no share between different ITests.
• Make ITest mandatory for each build. No, each Jenkins job has own configuration and parameters. Their interactions with ITests can be configured according to context at each job execution. Executing ITests is not mandatory.

Guide-level Explanation

After the resolution of this proposals, you will be able to:

• Trigger integration tests locally
• Trigger integration tests via on-demand job on Jenkins (optional)
• Trigger integration tests via master/maintenance job on Jenkins (mandatory)

and the following functionalities remain the same:

• Test logic
• Firefox versions

Reference-level Explanation

This is an implementation proposal (step-by-step):

1. Copy existing tests to NOS repository
2. Trigger them locally to ensure every test works
3. Trigger them via Jenkins. Ensure the Unresolved questions are addressed.
4. Add option to on-demand job to trigger itests. (At this step, we consider that the proposal is almost done)
5. Disable existing itests jobs (master)
6. Delete DSL script
7. Wait for next major release, so that maintenance branch has the same mechanism
8. Disable existing itests jobs (maintenance)

Drawbacks

Why should we not do this?

• Possibility to have dependency issue. We will handle dependency issue with care. Making dependency shortcut is not our goal. See Non Goals section.
• Prevent QA team's ongoing work (branch) to be continued. Before repository merge, they should be notified, so that all ongoing branches are pushed to GitHub. After repository merge, QA team's branches will be kept in NOS.
• Execution timing on tests. Need further discussion on this topic.

Rationale and Alternatives

None.

Unresolved Questions

• Can we use multiple Firefox versions in a single Jenkins build?
• How to keep commit history of Nuxeo ITests? ( Is it necessary?)