Uploaded image for project: 'Nuxeo Platform'
  1. Nuxeo Platform
  2. NXP-30802

Modify helm charts to allow for pull policy specification on busybox init containers

    XMLWordPrintable

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: helm-chart-3.0.8
    • Component/s: Helm Chart

      Description

      Problem: The busybox init containers being used for kafka, elastic, mongo and more do not specify an image pull policy. As a result, it defaults to Always. This is causing issues in AWS ever since DockerHub put limits on anonymous (free) image pulls, where the user is limited to 100 per hour. For every nuxeo tenant, the user has multiple api and worker pods, and for every pod there are 3 init containers. This means at a minimum they are pulling the busybox image 12 times per tenant. For larger clients, it's upwards of 90 times (if running 30 pods), which quickly puts us at our limit, particularly when we have dozens of tenants running in the same EKS cluster. When scaling up/down, redeploying, or doing EKS upgrades, this causes outages due to image pull issues, and we then have to wait 6 hours before the limit resets and we are allowed to pull images, which leads to customer outages for extended periods of time. GCP/GKE appears to be handling caching differently than AWS/EKS which may explain why we don't experience similar issues in GCP, but regardless, we need to address this issue to avoid unnecessary pulls and outages.

      Request: Modify helm chart for init containers to allow for image pull specification. Below chart snippet shows how it's allowed for the nuxeo container, but not for init containers. Update chart, and default values.yaml, with default of always, but which they can then override as IfNotPresent. Suggesting something called initImagePullSecrets, to differentiate from imagePullSecrets which is used for the nuxeo container, so that init containers and nuxeo containers can specify different policy values.

          spec:
            imagePullSecrets: \{{ toYaml . | nindent 6}}
            containers:
            - name: \{{ .Chart.Name }}
              image: "\{{ .Values.image.repository }}:\{{ .Values.image.tag }}"
              imagePullPolicy: \{{ .Values.image.pullPolicy }}
              env:
              volumeMounts:
            initContainers:
            - name: init-mongodb
              image: busybox
              command: ['sh', '-c', 'until nc -w1 \{{ .Values.mongodb.host }} \{{ .Values.mongodb.port }}; do echo "waiting for mongodb"; sleep 2; done;']
            - name: init-elasticsearch
              image: busybox
              command: ['sh', '-c', 'until nc -w1 \{{ .Values.elasticsearch.host }} \{{ .Values.elasticsearch.port }}; do echo "waiting for elastic"; sleep 2; done;']
            - name: init-kafka
              image: busybox
              command: ['sh', '-c', 'until nc -w1 \{{ .Values.kafka.host }} \{{ .Values.kafka.port }}; do echo "waiting for kafka"; sleep 2; done;']
      

        Attachments

          Issue Links

            Activity

              People

              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: