External CI

The goal is to migrate parts by parts from the current QA/CI hosted on https://qa.nuxeo.org.
This is now possible thanks to the previous work done on NXDRIVE-764.

This is required to lower the workload of the current QA/CI and benefit of public and fast PR statuses.

Note: the legacy CI will be kept for packaging purpose (for now).

Step 1

Move code quality checks and unit tests.
This will become a blocker status for any PR.

-> Done with NXDRIVE-2064.

Step 2

Move functional tests.
This will not become a blocker status for any PR. At least not at the beginning.

This part will be more complexe to achieve. This requires:

• A complete Nuxeo setup to handle sync work.
• A way of preventing timeouts as tests are currently running in about 2 hours (maximum, on Windows with 4 CPU).

Step 3

Move Windows integration tests.
This will become a blocker status for any PR.

But this also needs more work to:

• Ensure we can generate the Windows binary.

—

Providers

There is currently no real decision taken, but the external provider must at least:

• Dispose of GNU/Linux, macOS and Windows agents.
• Have VMs/machines with at least 4 CPUs.

Travis-CI

https://docs.travis-ci.com/user/reference/overview/

• Free for open-source projects (which is the case for Nuxeo Drive).
• Setup very easy.
• Job timeout can be workarounded.
• Has several macOS versions, which is a good fit for the packaging.

Cons:

• macOS agents are quite slow to start (sometimes up-to 10 minutes). Not really an issue.
• Docker cannot be used on macOS (which means no Nuxeo server).

Notes:

• Log size limit is 4 MB (#1382).

GitHub Actions

https://help.github.com/en/actions/reference/virtual-environments-for-github-hosted-runners

• Free for open-source projects (which is the case for Nuxeo Drive).
• Setup very easy.
• Integrated to GitHub, so it should be faster and more future-proof.
• Job timeout handled.

Cons:

• Only 1 supported version of macOS. It disqualifies it for packaging stuff and integrations tests.
• Only 2 processors.
• Docker cannot be used on macOS (which means no Nuxeo server).

CircleCI

https://circleci.com/docs/2.0/executor-types/

• Free for open-source projects (which is the case for Nuxeo Drive).
• Setup very easy.
• Job timeout are handled, but not ideally and maybe the travis-CI workaround would be better.

Cons:

• No macOS agents with the free plan.
• Only 2 processors for Linux VMs with the free plan.
• A lot of Python segfaults with the Linux agent.

AppVeyor

https://www.appveyor.com/docs/build-environment/

Pros:

• Free for open-source projects (which is the case for Nuxeo Drive).
• Setup very easy.

Cons:

• Only 2 supported versions of macOS. It disqualifies it for packaging stuff and integrations tests.
• Only 2 processors.
• Docker cannot be used on macOS (which means no Nuxeo server).