-
Type: Task
-
Status: Resolved
-
Priority: Minor
-
Resolution: Fixed
-
Affects Version/s: None
-
Fix Version/s: None
-
Component/s: Jenkins X
-
Tags:
-
Team:PLATFORM
-
Sprint:nxplatform 11.1.31, nxplatform 11.1.32
-
Story Points:3
Currently we're using the Jenkins X provided Docker registry to deploy images:
- jenkins
- builders
- nuxeo
See the complete list:
curl https://docker-registry.platform.dev.nuxeo.com/v2/_catalog | jq
This is an issue at least because:
- We're relying on an extra service whereas we already have/need a Nexus for the Maven artifacts.
- We quickly run out of disk space since there is no cleanup policy in the registry, see the frequent:
[2020-02-21T21:29:16.681Z] WARN[0032] error uploading layer to cache: failed to push to destination 10.63.253.90:5000/nuxeo/builder/cache:dd39ea3992046f41bfd389a6880c8992d4366883b0ee18861a740fc4085c506f: UNKNOWN: unknown error; map[Enclosed:map[Op:mkdir Path:/var/lib/registry/docker/registry/v2/repositories/nuxeo/builder/cache/_uploads/7d92c49a-66d8-42a4-bb28-3e8af22c727e Err:28] DriverName:filesystem] [2020-02-21T21:29:16.681Z] error pushing image: failed to push to destination 10.63.253.90:5000/nuxeo/builder:PR-3778-11.1-SNAPSHOT: UNKNOWN: unknown error; map[DriverName:filesystem Enclosed:map[Op:mkdir Path:/var/lib/registry/docker/registry/v2/repositories/nuxeo/builder/_uploads/12917975-2783-4b5e-b4a9-1f673b4acec3 Err:28]] [2020-02-21T21:29:21.957Z] time="2020-02-21T21:29:21Z" level=fatal msg="build failed: build failed: building [builder]: kaniko build for [builder]: waiting for pod to complete: condition error: pod already in terminal phase: Failed"
and logs in the Docker registry pod like:
mkdir /var/lib/registry/docker/registry/v2/blobs/sha256/ac/acd835ecf2999b95553b16575af538e2153e592dd41dc6a33204a3400ab2a933: no space left on device"
Meanwhile, we've implemented a script to purge the registry, that runs every night, see https://jenkins.platform.dev.nuxeo.com/job/nuxeo/job/purge-docker-registry/.
But this is not sufficient nor a proper solution.