Uploaded image for project: 'Nuxeo Platform'
  1. Nuxeo Platform
  2. NXP-31891

Benchmark job failing on Ingress API version - Upgrade Ingress NGINX Controller for K8s 1.23



    • Type: Bug
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2023.0, 2021.39
    • Component/s: CI/CD


      The benchmark job was failing while trying to install the nuxeo Helm chart:

      Error: unable to build kubernetes objects from release manifest: unable to recognize "": no matches for kind "Ingress" in version "networking.k8s.io/v1beta1"

      The Kubernetes cluster was upgraded on 18 May 2023 from 1.22.17-gke.7500 to 1.23.17-gke.1700:

      $ gcloud container operations list
      NAME                                                          TYPE            LOCATION    TARGET                     STATUS_MESSAGE  STATUS  START_TIME                      END_TIME
      operation-1684410779047-3559ae2f-cbcb-4d52-b7bf-bb5bdcb8594a  UPGRADE_MASTER  us-east1-b  jx-prod                                    DONE    2023-05-18T11:52:59.047578704Z  2023-05-18T11:57:55.419618162Z
      $ kubectl version --short
      Server Version: v1.23.17-gke.1700

      Thus, the Ingress beta API version removed in 1.22, though still supported in 1.22 by GKE, doesn't work anymore in 1.23.

      Since NXP-31102, fixed for the Nuxeo Helm chart 3.0.11, we should have the right API version depending on the Kubernetes server capabilities.
      Currently, the benchmark is relying on version 3.0.1 of the Nuxeo Helm chart. Let's upgrade to the latest version.

      Note: helm template falsely detects the API version capabilities, see https://github.com/helm/helm/issues/11809, which results in:

      helm template --set ingress.enabled=true https://chartmuseum.platform.dev.nuxeo.com/charts/nuxeo-3.0.13.tgz | grep ingress -A2
      # Source: nuxeo/templates/ingress.yaml
      apiVersion: networking.k8s.io/v1beta1
      kind: Ingress

      Yet, helm install in dry run outputs the expected value:

      helm install --dry-run --set ingress.enabled=true test https://chartmuseum.platform.dev.nuxeo.com/charts/nuxeo-3.0.13.tgz | grep ingress -A2
      # Source: nuxeo/templates/ingress.yaml
      apiVersion: networking.k8s.io/v1
      kind: Ingress


      The upper mentioned fix was working fine until the NGINX Ingress Controller pod was still running. Once restarted, the pod would never be ready, complaining in the logs about beta Ingress API version, definitively unsupported in 1.23.

      We've needed to upgrade the controller to the latest version.
      This was done by adding an ingress-nginx chart in platform-ci, installed in the ingress-nginx namespace, thus as a cluster-wide resource.
      It is only installed with the "production" environment, aka when merging a pull request on the master branch.


      • Defines a default Ingress class for the cluster.
      • Removed the in deprecation annotation "kubernetes.io/ingress.class: nginx" from the Ingress resources (jenkins, nexus, docker, chartmuseum).

      In the same way, we are now installing Cert ManagerĀ in the cert-manager namespace (cluster-wide resource) through the cert-manager chart in platform-ci, upgrading it to v1.12.x.
      It was originally installed manually in version v1.11.1, see NXBT-3716.


          Issue Links



              • Votes:
                0 Vote for this issue
                2 Start watching this issue


                • Created: