Uploaded image for project: 'Nuxeo ECM Build/Test Environment'
  1. Nuxeo ECM Build/Test Environment
  2. NXBT-348

Improve CI and Testing framework

    XMLWordPrintable

    Details

    • Tags:
    • Backlog priority:
      400

      Description

      Technical debt on Test

      Non Regression tests

      • rewrite Selenium tests in WebDriver

      => make them more modular and easier to debug

      • improve Nuxeo HTML layout to make tests easier

      => make tests more robust

      Funkload tests

      Currently only Ben can really fix a broken test.

      We should probably improve training and documentation so that not only Ben can fix them when needed.

      => migrate CMIS and Rest Tests in java

      Multi-DB / Multi-OS

      MultiDB and Multi-OS tests matrix have been added recently.

      There is still a lot of work to be done so that they all run :

      • fix unit tests on MSSQL and MySQL (h2, PGSQL et Oracle are ok)
      • Windows : there are still failed unit tests under windows
      • OpenJDK 7 : there are still failed unit tests under windows

      Non regression testing in Matrix should also be fixed but it's probably pointless until all unit tests are fixed.

      CI infrastructure speedup

      The goal is to make CI feedback quicker.
      We can not expect CI feedback to take a only few minutes :

      • build is big
      • we have a lot of tests in different configuration

      But there are some actions we can do to improve the feedback time.

      fix Nexus / Jenkins concurrency issue

      Nexus and Jenkins master share the same server and the same NAS.

      • Jenkins Master uses NAS storage to store the workspace (build and test results)
      • Nexus uses NAS to store the repo

      Monitoring shows that the server has a load average about 2 and that Jenkins or Nexus are faster when the other service is off.

      This means we should work on the global optimization of this Nexus/Jenkins setup : that will globalling improve speed of all CI :

      • for fetching / pushing artifacts
      • for storing results

      Based on Ben's remarks, the main lead for improvement seems to be the NFS share :

      • separate NASs for Jenkins / Nexus ?
      • use local storage instead of NAS ?
      • NFS / NAS tunning ?
      • ...

      GWT Build issue

      GWT build is really very slow and represent a big part of the global Nuxeo build time event if we only have a few (3 or 4) GWT modules.

      => This needs to be fixed, both for CI and for developpers

      Identified solutions include :

      • change profile to limit the number of target browser (meaning nb of permitation) during the standard Build
        ( we will still need to do the full build for releases)
      • disable GWT optimizations during standard builds
      • be sure to only trigger the GWT build when needed

      See http://stackoverflow.com/questions/1011863/how-do-i-speed-up-the-gwt-compiler

      NB : we can also extract GWT modules in Addons to help running GWT builds in parallel

      Slave build performances

      Currently the build on the Jenkins slaves is about the same time than on a dev laptop.

      => we can make better !

      It's true that a big part of the Jenkins slave are poor hardware (the dwarf).

      Before taking any decision, we should probably do some profiling to see exactly what are the real impacting factors :

      • test build on a bigger CPU (like Octo)
      • use RAMFS to speedup I/O
      • tweak compile options ?
      • test on big RackSpace VMs

      => based on that we will be able to define what kind of Build slaves we need

      (then we will have to see what we can host internaly and what must be external)

      CI Config

      Permanent checks

      Permanent checks currently don't wait for nuxeo-features to be built but won't raise relevant issues until nuxeo-features is built.
      In cases where there are a lot of commits, nuxeo-features build can take several hours.

      => need to change that

      So basically, we could have permanent checks be closer to the IT builds :

      • rebuild everything from zero
      • do build / no UT / distrib on a fast slave
      • dispatch non-reg tests on different hosts

      Addons

      Current distribution build depends on some addons: the one that are considered standard and used in distribs / MP packages.

      => Create a job that build all "standard" addons so that distribution does not have to wait for 20 jobs

      JBoss distrib

      We need to update JBoss distribution to align on tomcat/MP build.

      Tests to add

      There are still some part of the platform that are poorly / not enough tested

      Studio

      XXX Let Anahide list what she has in mind, but basically :

      • define a standard Studio project
      • run build against different targets
      • check result in a running Nuxeo on several target versions

      Admin Center / Marketplace

      We already have a webdriver test that tests the basis.

      We need to extend it so that we validate for example :

      • add CMF / uninstall DM
      • Install DAM / uninstall DAM + install CMF
      • ...

      XXX more here

        Attachments

          Issue Links

            Activity

              People

              • Votes:
                0 Vote for this issue
                Watchers:
                0 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  PagerDuty