ingress-nginx retirement: a Traefik and Gateway API pilot, then rollback

The ingress-nginx project has been retired by Kubernetes and no longer ships security patches. The controller still runs, but any vulnerability found from now on will not be fixed, so every cluster using it has to plan a migration path. This article explains how Gateway API differs from Ingress, how to run a Traefik pilot in parallel on production without changing DNS, how to keep the load balancer IP when switching controllers, the common mistakes during a pilot, and why a pilot can end in a rollback and still be a success.

1. Options for replacing ingress-nginx

The articles Ingress and NodeBalancer: getting traffic into a Kubernetes cluster and ingress-nginx and cert-manager: automatic HTTPS on Kubernetes described how traffic enters the cluster. Now that kubernetes/ingress-nginx is no longer developed, there are a few options:

Traefik is often tried first because it allows a soft transition: old sites keep using Ingress, new sites use Gateway API's HTTPRoute, all on one controller.

2. How Gateway API differs from Ingress

Gateway API is the API set that succeeds Ingress, and it is already GA (v1). Instead of one Ingress object carrying everything and pushing the rest into annotations, Gateway API splits things by role:

• GatewayClass: which controller handles it (Traefik, NGINX Gateway Fabric...). Equivalent to IngressClass.
• Gateway: the entry point, made of listeners (port, protocol, hostname, TLS certificate). The infrastructure team owns this object.
• HTTPRoute: the routing rules of each application, attached to a Gateway via parentRefs. Each site owns its own route.
• ReferenceGrant: allows cross-namespace references, for example a Gateway reading a TLS Secret in the site's namespace.

Moving a web chart to Gateway API means turning each ingress-nginx annotation into the corresponding mechanism:

The last two rows show the limits of standard Gateway API: body size and timeouts are not yet part of the standard, and each controller handles them differently. That part stays tightly coupled to the chosen controller.

3. Keeping the load balancer IP when switching controllers

On LKE, the NodeBalancer is tied to a Service of type LoadBalancer, not to the software behind it. Uninstalling the ingress-nginx Helm release deletes the Service, the Cloud Controller Manager deletes the NodeBalancer along with it, and the IP is gone. Installing a new controller with default settings gets a new IP. Both paths lead to changing DNS en masse.

How to keep the IP: do not delete the old Service, install the new controller with a ClusterIP Service, then switch the old Service's selector to the new controller's pods. Three common mistakes:

1. Wrong field name in the chart. Some Traefik chart versions put the Service type under service.spec.type, not service.type. Set the wrong one and Helm says nothing, the Service stays a LoadBalancer, and a new load balancer is created, which costs money. Always run helm show values before using --set.
2. Different namespace. A Service can only select pods in its own namespace. Install the new controller in another namespace and, after the patch, the Service has 0 endpoints and the domains lose traffic immediately.
3. A merge patch does not remove old keys. kubectl patch --type merge merges each key in selector. A leftover old key makes the Service require both the old and the new labels, and again it has 0 endpoints. You must set the old key to null in the same command.

kubectl patch svc ingress-nginx-controller -n <ns> --type merge -p '{
  "spec": {
    "selector": {
      "app.kubernetes.io/component": null,
      "app.kubernetes.io/name": "traefik",
      "app.kubernetes.io/instance": "traefik"
    },
    "ports": [
      {"name": "http",  "port": 80,  "targetPort": 8000, "protocol": "TCP"},
      {"name": "https", "port": 443, "targetPort": 8443, "protocol": "TCP"}
    ]
  }
}'
kubectl describe svc ingress-nginx-controller -n <ns> | grep -E "Selector|Endpoints"

Rollback means patching the selector back to the ingress-nginx pods, which takes effect within seconds if the old pods are still running in parallel. After every patch, check that Endpoints is not empty before calling it done. "patched" does not mean traffic has reached the right place. Rehearse this whole procedure on a test cluster first.

4. Running a Gateway API pilot in parallel on production

The cheapest and safest way to try it:

1. Install the Gateway API CRDs (standard channel) and the controller's CRDs. CRDs are a new API group and do not affect the running Ingress objects.
2. Install the pilot controller as a separate release, in a separate namespace, with its own LoadBalancer Service and a new IP.
3. Do not change DNS. Visitors still go through ingress-nginx. The pilot is tested by calling the new IP directly with a Host header: curl -H "Host: site-a.example.com" http://<ip-pilot>/.
4. Pick a few representative sites: at least one with TLS and one without, with each pilot site running in its own namespace.
5. Save all values, manifests and Helm history of each site before touching anything.

The first common error when creating the Gateway: an HTTPS listener with empty certificateRefs is rejected (certificateRefs or options must be specified when mode is Terminate). The fix is to attach a self-signed certificate as a placeholder.

Storage errors when running in parallel

• Duplicate PersistentVolume names. A PV is a cluster-level resource. If its name comes from fullnameOverride and you reuse the real site's values unchanged, the pilot PV collides with the production PV and Helm refuses. This static PV design is covered in Helm charts with static NFS PersistentVolumes, claimRef and three probe types.
• Renaming changes the NFS path too, the new directory does not exist yet, and the mount fails.
• A helm install that times out keeps running in the background and may overwrite the patch you just made. Wait for the release to reach a final state before editing by hand.
• Deleting a PVC hangs because a pod stuck in init still holds it (the kubernetes.io/pvc-protection finalizer). Scale the Deployment to 0 first, then delete.
• Init containers run against real data. If the pilot shares the production database, php artisan migrate --force in the init container will change the real site's schema when the pilot image is newer. The pilot must use exactly the image currently running, or disable migrations.

A global redirect makes non-TLS sites return 404

Setting the HTTP to HTTPS redirect once on the entrypoint relies on the assumption that every site is the same. That assumption holds for timeouts, but not for TLS: a site without TLS gets redirected to https://, where there is no route, and returns 404. The redirect belongs to each site: the chart only creates the https-redirect HTTPRoute when tls.enabled is true.

Two small errors

• Helm renders large numbers such as 134217728 (128 MiB) as 1.34217728e+08, and the API rejects it because the field needs an integer. Fix it with {{ .Values.maxBodySizeBytes | int64 }}.
• cert-manager may issue new certificates for pilot site domains, because the HTTP-01 challenge still goes through the production controller. It does no harm, but it uses up Let's Encrypt's weekly duplicate certificate limit.

5. Gateway API certificateRefs is an atomic field

This is often the issue that decides the fate of the pilot. In Gateway API, a listener's certificateRefs list is an atomic field: whoever writes last replaces the whole list, with no merging. Two independent Helm releases writing to a shared Gateway will wipe out each other's certificates, and sites end up served with the controller's default certificate.

A temporary fix is to take Gateway write access away from the site chart: each site only labels its own TLS Secret, a single script collects every Secret with that label and rewrites certificateRefs, relying on resourceVersion to retry on conflict, with a CronJob as a safety net. It works, but adds a single point of failure for every hostname. The proper solution is ListenerSet, which lets each site declare its own listener and certificate without editing the Gateway. If the controller you chose does not support ListenerSet yet, that is a legitimate reason to stop the pilot.

6. Rollback and the cleanup list

Because production was never cut over, rollback is just removing the pilot namespaces and the pilot controller and putting values back where they were, with no disruption. But a rollback needs a cleanup list:

• CRDs. Helm does not delete CRDs when uninstalling a release. The Gateway API CRDs can be reused for the next step; the pilot controller's own CRDs need to be removed deliberately.
• PVs, PVCs and NFS directories of the pilot sites.
• Side effects on pipelines. If you isolated pilot sites by moving their values files to another directory so the deploy pipeline would not overwrite them, the pipeline will also skip those very production sites. After the rollback, check that these sites have received the latest build again.
• The load balancer and IP of the pilot controller.

7. Checklist for moving from ingress-nginx to Gateway API

• List every ingress-nginx annotation in use and its replacement mechanism.
• Verify the "every site is the same" assumption against real values, key by key.
• Read the semantics of Gateway API fields (atomic or merge) before designing the chart.
• Parallel pilot: separate IP, separate namespace, unchanged DNS, testing with the Host header.
• The pilot uses exactly the running image, never letting migrations change the production schema.
• A procedure for keeping the load balancer IP, rehearsed on a test cluster.
• A cleanup list for rollback.

The concept page Gateway API on kubernetes.io is a good starting point. The other articles in the series are in the Sudo technical blog.

Next article

NGINX Gateway Fabric: ListenerSet and cert-manager 1.21 on LKE. One Gateway, one load balancer, and each site declares its own certificate with ListenerSet. It is also the final article of the series.