Kubernetes monitoring with kube-prometheus and Grafana

Once the cluster has ingress, automatic certificates and applications running smoothly, questions like "are the nodes getting full", "which pod uses the most RAM" or "was anything unusual last week" can still only be answered by typing kubectl and guessing. This guide sets up Kubernetes monitoring with kube-prometheus-stack: what the toolkit contains, how a ServiceMonitor works, configuration for an LKE cluster, how to expose Prometheus and Grafana through an Ingress with basic auth, and the common configuration mistakes.

1. Why Kubernetes monitoring is needed early

On Cloud VPS, each machine is an easy unit to picture: SSH in, run htop, read the logs. In a cluster, workloads drift between nodes, pods die and are recreated with new names, and history disappears with the pod. kubectl top only tells you about right now, not yesterday. Kubernetes monitoring answers three groups of questions that kubectl cannot:

• Resources over time: CPU, RAM, disk and network for each node and each pod over many days, so you know how much each pod really needs.
• Object state: which pods keep restarting, which Deployments are missing replicas, which PVCs are almost full.
• Declared versus actual: what percentage of a node the total requests already claim, compared with real usage.

The third question is especially important. The article Building the first LKE cluster: node pools, kubeconfig and early choices explained that the scheduler places pods by requests. Without Kubernetes monitoring, every request value is a guess.

2. What kube-prometheus-stack contains

Instead of assembling each piece of a Kubernetes monitoring system yourself, you can use the Prometheus community's kube-prometheus-stack Helm chart. The chart packages the kube-prometheus project as a single release:

In Kubernetes monitoring, these two exporters are easy to confuse. node-exporter looks from the server's point of view: how much RAM does this node have left. kube-state-metrics looks from Kubernetes' point of view: this Deployment wants 1 replica but has 0. How much CPU a pod uses is the job of cAdvisor, built into kubelet, which the chart also configures for scraping.

Prometheus Operator and CRDs

Prometheus Operator brings Kubernetes monitoring into the operator model: instead of editing a prometheus.yml file, you declare Kubernetes objects and the operator generates the configuration. The chart installs CRDs such as Prometheus, Alertmanager, AlertmanagerConfig, ServiceMonitor, PodMonitor, Probe, PrometheusRule, PrometheusAgent, ScrapeConfig and ThanosRuler. Full documentation is on the Prometheus Operator site.

ServiceMonitor: declaring what to scrape

A ServiceMonitor tells the operator: "scrape the Services with this label, on the port with this name, at this path". An example for an application with a /metrics endpoint:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: my-app
  namespace: monitoring
  labels:
    release: <release-name>        # so the chart's Prometheus selects this ServiceMonitor
spec:
  namespaceSelector:
    matchNames: [my-app]
  selector:
    matchLabels:
      app: my-app
  endpoints:
  - port: metrics
    path: /metrics
    interval: 30s

The most common trap when you start Kubernetes monitoring with this chart is the release label. With the chart's default values, Prometheus only selects ServiceMonitors carrying a release label equal to the Helm release name. Without that label, the ServiceMonitor is still created successfully but never shows up in the target list. If the Laravel application has no metrics endpoint of its own yet, the chart's built-in ServiceMonitors (node, kubelet, kube-state-metrics, CoreDNS, Prometheus itself) are enough for the first stage.

3. Kubernetes monitoring configuration for an LKE cluster

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update
kubectl create namespace monitoring
helm install <release-name> prometheus-community/kube-prometheus-stack \
  --namespace monitoring --version <version> \
  -f values-monitoring.yaml

A sample values file for the Kubernetes monitoring system, with the ingress part trimmed (see section 4):

prometheus:
  prometheusSpec:
    routePrefix: /
    externalUrl: https://monitoring.example.com/prometheus
    retention: 10d
    resources:
      requests: {cpu: 200m, memory: 1Gi}
      limits:   {memory: 2Gi}
    storageSpec:
      volumeClaimTemplate:
        spec:
          storageClassName: linode-block-storage-retain
          resources:
            requests:
              storage: 10Gi

alertmanager:
  enabled: false          # enable once you know what to alert on

grafana:
  persistence:
    enabled: true
    storageClassName: linode-block-storage-retain
    size: 10Gi
  grafana.ini:
    server:
      domain: monitoring.example.com
      root_url: "%(protocol)s://%(domain)s/grafana/"

# The control plane is managed by LKE and cannot be scraped
kubeEtcd:
  enabled: false
kubeControllerManager:
  enabled: false
kubeScheduler:
  enabled: false

Three decisions worth noting:

• Disable scraping of etcd, the controller manager and the scheduler. The LKE control plane is operated by Linode, and these components do not expose metrics ports to users. Left at the defaults, Prometheus has three target groups that are always down, and the bundled rules raise false alarms.
• Prometheus and Grafana get their own volumes with a Retain StorageClass, as in Volume, PersistentVolume, PVC and StorageClass on Kubernetes. Without a volume, a Prometheus restart loses all Kubernetes monitoring history, and a Grafana restart loses custom dashboards. Capacity has to grow with retention and the number of series.
• Alertmanager can be disabled at first, while you still need to look at the data to decide what to alert on. But that is only a temporary step: without alerts, the system is only useful when someone opens Grafana to look.

4. Protecting Prometheus and Grafana with basic auth through Ingress

Prometheus has no login. Anyone who reaches its UI can see every metric, including namespace names, pod names and each site's domain. Grafana has its own login, but an extra layer in front is not overkill. You can expose both Kubernetes monitoring UIs on one domain, separated by path, and protect them with ingress-nginx basic auth:

# create the htpasswd file and Secret in the monitoring namespace
htpasswd -c auth <user>
kubectl create secret generic basic-auth --from-file=auth -n monitoring
rm auth                              # do not leave this file in the repo

prometheus:
  ingress:
    enabled: true
    ingressClassName: nginx
    annotations:
      nginx.ingress.kubernetes.io/rewrite-target: /$2
      nginx.ingress.kubernetes.io/auth-type: basic
      nginx.ingress.kubernetes.io/auth-secret: basic-auth
      nginx.ingress.kubernetes.io/auth-realm: 'Authentication Required'
    hosts:
    - monitoring.example.com
    paths:
    - /prometheus(/|$)(.*)
    tls:
    - secretName: monitoring-tls
      hosts:
      - monitoring.example.com
# grafana.ingress works the same way, path /grafana(/|$)(.*), same secretName monitoring-tls

The regex path /prometheus(/|$)(.*) combined with rewrite-target: /$2 strips the /prometheus prefix before forwarding to the pod. So Prometheus still serves at / (routePrefix: /), while externalUrl tells it the public address so it generates correct links. Grafana uses the same approach, with a root_url that includes /grafana/. The basic auth mechanism is described in the ingress-nginx Basic Authentication example: the file must be named auth, because ingress-nginx reads exactly that key in the Secret.

The certificate for the Kubernetes monitoring domain follows the approach in ingress-nginx and cert-manager: automatic HTTPS on Kubernetes: an explicit Certificate writing to the monitoring-tls Secret, using the cluster's shared ClusterIssuer.

5. Common configuration mistakes in Kubernetes monitoring

None of the mistakes below shows up in red anywhere; they only surface when you reread the values line by line:

• An annotation pointing to an issuer that does not exist. If an Ingress has both a cert-manager.io/cluster-issuer annotation with the wrong name and a tls block, ingress-shim generates a second Certificate writing to the same TLS Secret, and that Certificate never becomes ready. HTTPS still works thanks to the explicit Certificate, so the error never shows in the browser. If you have written an explicit Certificate, drop the annotation.
• Two Ingresses with the same host and different TLS Secrets. The Grafana Ingress points to a Secret that no Certificate creates. For the same domain, share one Secret so you do not depend on how ingress-nginx merges configuration.
• A separate ClusterIssuer just for monitoring. A ClusterIssuer already works for every namespace; a copy that differs only in the ACME account key Secret is redundant and invites name confusion.
• The auth file in Git. Even though it only holds hashes, an htpasswd file should not go into the repo: weak hashes can be cracked, and anyone reading the repo learns the user name.
• Monitoring sharing nodes with websites, without resources. Prometheus eats RAM and competes for resources with the very pods it is measuring.
• A ServiceMonitor missing the release label, as in section 2.

6. Where monitoring should run

A Kubernetes monitoring system should not die at the same time as the thing it monitors. A website under attack drives a node out of RAM, and just when you need the data most, the Prometheus on that same node gets OOM-killed. Put Prometheus, Grafana and the operator on the system node pool, following the idea of splitting pools by role from the cluster-building article, and set explicit requests and limits. To do that you need to understand requests, limits, QoS classes and nodeAffinity on the lke.linode.com/pool-id label, the subject of the next article.

Once you have Kubernetes monitoring, the way you tune resources changes too: CPU and RAM requests for MySQL or the web container should only be lowered or raised after you have seen real usage over many days in Grafana, not changed on gut feeling.

7. Kubernetes monitoring checklist

1. kube-prometheus-stack is pinned to a version, and the values are in Git (without secrets).
2. Scraping is disabled for control plane components the managed service does not expose.
3. Prometheus and Grafana have Retain volumes, with matching retention and capacity.
4. The UIs are exposed over HTTPS with basic auth, using one shared TLS Secret per host.
5. Custom ServiceMonitors carry the correct release label.
6. Monitoring runs in the system pool, with resources set.
7. There is a plan to enable Alertmanager with a minimal set of alerts.

Setting up Kubernetes monitoring with kube-prometheus-stack is much faster than assembling it yourself: one Helm command gives you Prometheus, exporters, dashboards and a large set of ready-made rules. The effort goes into adapting it to your environment and keeping the configuration of the Kubernetes monitoring system itself correct. When you need to build your own dashboards, the Grafana documentation is the place to look things up.

Next article

Requests and limits, QoS and nodeAffinity: placing pods on the right nodes. Reading the numbers in Grafana to reset resources, understanding the three QoS classes, and forcing each group of pods onto the right node pool.