Uploaded image for project: 'Nuxeo ECM Build/Test Environment'
  1. Nuxeo ECM Build/Test Environment
  2. NXBT-3255

Use JX internal Nexus as Docker repository in the platform team

    XMLWordPrintable

    Details

    • Type: Task
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: Jenkins X
    • Team:
      PLATFORM
    • Sprint:
      nxplatform 11.1.31, nxplatform 11.1.32
    • Story Points:
      3

      Description

      Currently we're using the Jenkins X provided Docker registry to deploy images:

      • jenkins
      • builders
      • nuxeo

      See the complete list:

      curl https://docker-registry.platform.dev.nuxeo.com/v2/_catalog | jq
      

      This is an issue at least because:

      • We're relying on an extra service whereas we already have/need a Nexus for the Maven artifacts.
      • We quickly run out of disk space since there is no cleanup policy in the registry, see the frequent:
        [2020-02-21T21:29:16.681Z] WARN[0032] error uploading layer to cache: failed to push to destination 10.63.253.90:5000/nuxeo/builder/cache:dd39ea3992046f41bfd389a6880c8992d4366883b0ee18861a740fc4085c506f: UNKNOWN: unknown error; map[Enclosed:map[Op:mkdir Path:/var/lib/registry/docker/registry/v2/repositories/nuxeo/builder/cache/_uploads/7d92c49a-66d8-42a4-bb28-3e8af22c727e Err:28] DriverName:filesystem] 
        [2020-02-21T21:29:16.681Z] error pushing image: failed to push to destination 10.63.253.90:5000/nuxeo/builder:PR-3778-11.1-SNAPSHOT: UNKNOWN: unknown error; map[DriverName:filesystem Enclosed:map[Op:mkdir Path:/var/lib/registry/docker/registry/v2/repositories/nuxeo/builder/_uploads/12917975-2783-4b5e-b4a9-1f673b4acec3 Err:28]]
        [2020-02-21T21:29:21.957Z] time="2020-02-21T21:29:21Z" level=fatal msg="build failed: build failed: building [builder]: kaniko build for [builder]: waiting for pod to complete: condition error: pod already in terminal phase: Failed"
        

        and logs in the Docker registry pod like:

        mkdir /var/lib/registry/docker/registry/v2/blobs/sha256/ac/acd835ecf2999b95553b16575af538e2153e592dd41dc6a33204a3400ab2a933: no space left on device"
        

      Meanwhile, we've implemented a script to purge the registry, that runs every night, see https://jenkins.platform.dev.nuxeo.com/job/nuxeo/job/purge-docker-registry/.
      But this is not sufficient nor a proper solution.

        Attachments

          Issue Links

            Activity

              People

              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0 minutes
                  0m
                  Logged:
                  Time Spent - 1 week, 2 days, 6 hours
                  1w 2d 6h