The benchmark job was failing while trying to install the nuxeo Helm chart:
Error: unable to build kubernetes objects from release manifest: unable to recognize "": no matches for kind "Ingress" in version "networking.k8s.io/v1beta1"
The Kubernetes cluster was upgraded on 18 May 2023 from 1.22.17-gke.7500 to 1.23.17-gke.1700:
$ gcloud container operations list NAME TYPE LOCATION TARGET STATUS_MESSAGE STATUS START_TIME END_TIME operation-1684410779047-3559ae2f-cbcb-4d52-b7bf-bb5bdcb8594a UPGRADE_MASTER us-east1-b jx-prod DONE 2023-05-18T11:52:59.047578704Z 2023-05-18T11:57:55.419618162Z ... $ kubectl version --short ... Server Version: v1.23.17-gke.1700
Thus, the Ingress beta API version removed in 1.22, though still supported in 1.22 by GKE, doesn't work anymore in 1.23.
Since NXP-31102, fixed for the Nuxeo Helm chart 3.0.11, we should have the right API version depending on the Kubernetes server capabilities.
Currently, the benchmark is relying on version 3.0.1 of the Nuxeo Helm chart. Let's upgrade to the latest version.
Note: helm template falsely detects the API version capabilities, see https://github.com/helm/helm/issues/11809, which results in:
helm template --set ingress.enabled=true https://chartmuseum.platform.dev.nuxeo.com/charts/nuxeo-3.0.13.tgz | grep ingress -A2 # Source: nuxeo/templates/ingress.yaml apiVersion: networking.k8s.io/v1beta1 kind: Ingress
Yet, helm install in dry run outputs the expected value:
helm install --dry-run --set ingress.enabled=true test https://chartmuseum.platform.dev.nuxeo.com/charts/nuxeo-3.0.13.tgz | grep ingress -A2 # Source: nuxeo/templates/ingress.yaml apiVersion: networking.k8s.io/v1 kind: Ingress
--------------------------------------------------------------------------------------------------------------------
The upper mentioned fix was working fine until the NGINX Ingress Controller pod was still running. Once restarted, the pod would never be ready, complaining in the logs about beta Ingress API version, definitively unsupported in 1.23.
We've needed to upgrade the controller to the latest version.
This was done by adding an ingress-nginx chart in platform-ci, installed in the ingress-nginx namespace, thus as a cluster-wide resource.
It is only installed with the "production" environment, aka when merging a pull request on the master branch.
Notes:
- Defines a default Ingress class for the cluster.
- Removed the in deprecation annotation "kubernetes.io/ingress.class: nginx" from the Ingress resources (jenkins, nexus, docker, chartmuseum).
In the same way, we are now installing Cert ManagerĀ in the cert-manager namespace (cluster-wide resource) through the cert-manager chart in platform-ci, upgrading it to v1.12.x.
It was originally installed manually in version v1.11.1, see NXBT-3716.