Improve CI and Testing framework

Technical debt on Test

Non Regression tests

• rewrite Selenium tests in WebDriver

=> make them more modular and easier to debug

• improve Nuxeo HTML layout to make tests easier

=> make tests more robust

Funkload tests

Currently only Ben can really fix a broken test.

We should probably improve training and documentation so that not only Ben can fix them when needed.

=> migrate CMIS and Rest Tests in java

Multi-DB / Multi-OS

MultiDB and Multi-OS tests matrix have been added recently.

There is still a lot of work to be done so that they all run :

• fix unit tests on MSSQL and MySQL (h2, PGSQL et Oracle are ok)
• Windows : there are still failed unit tests under windows
• OpenJDK 7 : there are still failed unit tests under windows

Non regression testing in Matrix should also be fixed but it's probably pointless until all unit tests are fixed.

CI infrastructure speedup

The goal is to make CI feedback quicker.
We can not expect CI feedback to take a only few minutes :

• build is big
• we have a lot of tests in different configuration

But there are some actions we can do to improve the feedback time.

fix Nexus / Jenkins concurrency issue

Nexus and Jenkins master share the same server and the same NAS.

• Jenkins Master uses NAS storage to store the workspace (build and test results)
• Nexus uses NAS to store the repo

Monitoring shows that the server has a load average about 2 and that Jenkins or Nexus are faster when the other service is off.

This means we should work on the global optimization of this Nexus/Jenkins setup : that will globalling improve speed of all CI :

• for fetching / pushing artifacts
• for storing results

Based on Ben's remarks, the main lead for improvement seems to be the NFS share :

• separate NASs for Jenkins / Nexus ?
• use local storage instead of NAS ?
• NFS / NAS tunning ?
• ...

GWT Build issue

GWT build is really very slow and represent a big part of the global Nuxeo build time event if we only have a few (3 or 4) GWT modules.

=> This needs to be fixed, both for CI and for developpers

Identified solutions include :

• change profile to limit the number of target browser (meaning nb of permitation) during the standard Build
  ( we will still need to do the full build for releases)
• disable GWT optimizations during standard builds
• be sure to only trigger the GWT build when needed

See http://stackoverflow.com/questions/1011863/how-do-i-speed-up-the-gwt-compiler

NB : we can also extract GWT modules in Addons to help running GWT builds in parallel

Slave build performances

Currently the build on the Jenkins slaves is about the same time than on a dev laptop.

=> we can make better !

It's true that a big part of the Jenkins slave are poor hardware (the dwarf).

Before taking any decision, we should probably do some profiling to see exactly what are the real impacting factors :

• test build on a bigger CPU (like Octo)
• use RAMFS to speedup I/O
• tweak compile options ?
• test on big RackSpace VMs

=> based on that we will be able to define what kind of Build slaves we need

(then we will have to see what we can host internaly and what must be external)

CI Config

Permanent checks

Permanent checks currently don't wait for nuxeo-features to be built but won't raise relevant issues until nuxeo-features is built.
In cases where there are a lot of commits, nuxeo-features build can take several hours.

=> need to change that

So basically, we could have permanent checks be closer to the IT builds :

• rebuild everything from zero
• do build / no UT / distrib on a fast slave
• dispatch non-reg tests on different hosts

Addons

Current distribution build depends on some addons: the one that are considered standard and used in distribs / MP packages.

=> Create a job that build all "standard" addons so that distribution does not have to wait for 20 jobs

JBoss distrib

We need to update JBoss distribution to align on tomcat/MP build.

Tests to add

There are still some part of the platform that are poorly / not enough tested

Studio

XXX Let Anahide list what she has in mind, but basically :

• define a standard Studio project
• run build against different targets
• check result in a running Nuxeo on several target versions

Admin Center / Marketplace

We already have a webdriver test that tests the basis.

We need to extend it so that we validate for example :

• add CMF / uninstall DM
• Install DAM / uninstall DAM + install CMF
• ...

XXX more here