Phantom 100% CPU from the hypervisor: when top does not tell the truth

An unpleasant situation on Cloud VPS: Grafana shows several machines at "CPU close to 100%" at the same time, or losing monitoring data entirely, including the balancer, the database machine and the minions. Traffic has not increased, there is no sign of errors in the application or MySQL, and a few minutes later everything returns to normal on its own. The culprit is not inside the virtual machine: the whole group of machines was held by the hypervisor. This guide shows how to prove that with data, and explains why top, Grafana's CPU Busy formula and even the steal time metric all fail to show the truth in this case.

Symptoms

• Some machines show CPU Busy close to 100%, and load spikes.
• Other machines show a broken graph (a GAP) for a few minutes, then jump to nearly 100% right before returning to normal.
• The affected machines have different roles, but show the same behavior within exactly the same time window.
• There may be misleading metrics as well, such as 100% SWAP on a minion.

Looking at the graphs, the first instinct is to suspect a DDoS or a heavy MySQL query. Both are wrong, and hardly anyone thinks of the hypervisor right away.

Investigation process

Step 1: is web traffic spiking?

Count requests per minute on the balancer around the incident window. Be careful with globs: if each site writes its own log to <domain>.access (without a .log suffix), the pattern /var/log/nginx/*access*.log only reads the default file and returns an absurdly small number. Count across every file, including compressed ones:

zcat -f /var/log/nginx/*.access* \
  | awk '{print substr($4, 2, 17)}' | sort | uniq -c | tail -30

The typical sign of a held machine: requests per minute drop instead of rising during the incident (the balancer slows down sharply), then arrive in a burst when it is released. There are no signs of an attack.

Step 2: read the raw CPU counters instead of the graph

The CPU Busy panel in the Node Exporter Full dashboard uses this formula:

1 - rate(node_cpu_seconds_total{mode="idle"}[1m])

Query Prometheus directly for the raw values of the idle counter on an affected machine:

node_cpu_seconds_total{mode="idle", instance="balancer:9100"}[10m]

On a 1 vCPU machine with a 30-second scrape, each sample normally adds about 20 seconds of idle. When the hypervisor holds the machine, the idle counter barely moves across several consecutive samples. rate(idle) is 0, so the formula returns 100%. In reality, the hypervisor is barely letting the virtual machine's vCPU run at all.

Step 3: look for traces in the kernel log

The journalctl -k output on the affected machines usually contains the same kind of warning:

kernel: clocksource: Long readout interval, skipping watchdog check:
    cs_nsec: 2803125000 wd_nsec: 2803118000

The kernel clock watchdog read twice several seconds apart, while the normal interval is only about half a second. Between those two reads, the virtual machine's vCPU received no cycles at all. This is a familiar sign of a virtual machine paused by the hypervisor, or stuck on I/O in the storage layer underneath.

Step 4: build a timeline from cron

Tasks that run every minute, such as Laravel's schedule:run, normally finish within a few seconds. The journal of a held machine shows a very distinctive pattern:

T+0s     CRON[...]: session opened for user www-data
T+59s    CRON[...]: session opened for user www-data
T+103s   CRON[...]: session opened for user www-data
T+244s   CRON[...]: session opened for user www-data
T+281s   CRON[...]: session closed   ← all four sessions close together
T+281s   CRON[...]: session closed
T+282s   CRON[...]: session closed
T+286s   CRON[...]: session closed

The sessions open at intervals over several minutes but all close within a few seconds. The machine did not slow down gradually: it was held by the hypervisor, then released all at once. On some machines node_exporter could not even answer Prometheus, so the graph shows a gap instead of hitting 100%.

Step 5: compare across the cluster to pin it on the hypervisor

If the hypervisor is to blame, machines on the same physical host are affected together and the others are not. Use a different query: add up every CPU mode (user, system, idle, iowait, steal…) and divide by the number of vCPUs. Normally this total is close to 1, because every real second of a vCPU has to be recorded by the kernel in some mode.

# share of CPU time recorded by the kernel (normally ≈ 1)
sum by (instance) (rate(node_cpu_seconds_total[1m]))
  /
count by (instance) (node_cpu_seconds_total{mode="idle"})

# steal time: time taken by the hypervisor that the guest kernel knows about
rate(node_cpu_seconds_total{mode="steal"}[1m])

When the two groups separate this cleanly, even though they share the same network range and run the same kinds of services, the only explanation is that the affected group sits on the same physical host, meaning the same hypervisor, or on the same storage pool.

The most notable point is the steal column. Steal time is the time a vCPU is ready to run but the hypervisor is giving the real CPU to another machine. When a virtual machine is paused outright, the time taken away usually does not go into steal, nor into iowait. It disappears from every counter.

Why top, vmstat and mpstat cannot see the hypervisor

top, vmstat and mpstat all read /proc/stat, the same counters node_exporter reads. Their %st column is calculated over the total recorded time, not over real time. If in 30 real seconds the kernel only managed to record a little user, system and softirq time and almost no idle, top shows mostly us and sy, 0% id and only about 1% st. In other words, "CPU busy, no steal". While the machine is held, the SSH session running top freezes too. To see an incident caused by the hypervisor, you have to look from another machine, such as a Prometheus server outside the affected group.

Root cause: the hypervisor stopped handing out CPU

The group of virtual machines was stopped, or nearly stopped, from receiving CPU by the hypervisor, all at the same time. When they were released, cron, timers, I/O and requests piled up and ran together, creating a very short real load spike, and then everything returned to normal.

Only the provider can see the physical host's logs. The most common hypothesis when an incident lands on a round hour and the whole group is affected: a scheduled task at the host level, such as a volume snapshot, a virtual machine backup or a live migration, freezing I/O or vCPUs for longer than usual.

Two wrong leads worth recording:

• 100% SWAP: a small swapfile that was already full beforehand while plenty of RAM is still available is a static state, unrelated to the incident. Enlarge the swapfile so it does not create noise next time.
• "Clock skew, the server has the wrong time": a common response when people see the word clocksource in the log. Checking timedatectl (System clock synchronized: yes) and comparing timestamps between machines rules it out. The Long readout interval line is a consequence of the vCPU being held, not the cause. Many independent machines also cannot all have "the wrong time" for exactly a few minutes and then recover together.

Remediation

During the incident: usually no intervention is needed, the system recovers on its own. What matters is not doing the wrong thing: do not restart MySQL, do not block IPs, do not optimize queries for an incident that lives in the hypervisor.

After the incident: send the provider an evidence package that includes:

• The list of affected machines and the time window (state the time zone clearly).
• The clocksource line with cs_nsec from the kernel log.
• The cron timeline showing the sessions closing together.
• The comparison table of the two machine groups with their recorded time ratio.

Include a request to check the hypervisor logs of the physical host and the snapshot/backup schedule around that time window, and to move those tasks to off-peak hours.

Prevention checklist

• Add a "recorded CPU time" panel next to CPU Busy, using the query from step 5. A value far below 1 on many machines at once is a sign of the hypervisor, not of the application.
• Alert by group, not by machine: when three or more machines have a ratio below 0.5 in the same minute, send one "suspected hypervisor incident" alert instead of a flood of high CPU alerts.
• Do not trust steal time alone. Steal is useful when a host oversubscribes its CPU. When a virtual machine is paused outright, steal may barely rise.
• Run Prometheus outside the affected zone, so it can still record GAPs and node_exporter timeouts, which are signals in their own right.
• When you see 100% CPU across the board: count requests in every log file, read the raw counters, search for clocksource in journalctl -k, check whether cron sessions close together, and compare with the group of healthy machines.

Lessons learned

• CPU Busy derived from idle is an inference, not a measurement. When the hypervisor pauses a virtual machine, "no idle" does not mean "busy".
• Double-check the commands you use to rule out a hypothesis. A wrong glob produces a wrong number, even if the conclusion sometimes happens to be right.
• A side-by-side comparison is the strongest evidence when you need to prove the fault lies in the provider's hypervisor: two groups of machines on the same network range, one broken and one not.
• Distinguish it from real 100% CPU. In 100% CPU on MySQL: waiting locks and missing indexes, the CPU was really busy because queries scanned many rows and waited on locks. Here there is no web load, and the whole group of machines is "busy" for exactly the same period.

How to work through each layer during an investigation is covered in Incident investigation across layers: from balancer to database and Kubernetes. The meaning of each CPU mode in /proc/stat, including steal, is described in the Linux kernel /proc documentation. node_exporter metrics are covered in the Prometheus node exporter guide.

Next article

WAF false positive and a duplicate rule ID that took down a balancer. One Coraza exception for one site, one reused rule ID, and every site on the balancer stopped serving at once.