Low-cost self-healing: detecting and restarting dead services

A small operations team cannot sit and watch dozens of Cloud VPS instances all day. Nginx or PHP-FPM dies at midnight, a queue worker falls into the FATAL state, a disk slowly fills up: these incidents recur regularly, and the fix is almost always the same. This guide shows how to build a low-cost self-healing layer: a few scheduled Jenkins jobs that SSH into each server, check service status, automatically restart whatever has died, and report the result to Slack. It covers job design, a sample script with safety limits, and most importantly, how to keep automatic fixes from hiding the root cause.

1. What self-healing is, and why do it on a budget

Self-healing is a system's ability to detect a broken state and bring itself back to the correct state without human intervention. In large ecosystems, an orchestrator takes care of this; Kubernetes, for example, recreates a container when a probe fails. A Cloud VPS cluster has no orchestrator: each service is a systemd unit or a supervisor program on a specific server. The self-healing part has to be built from three components:

1. Detection: a scheduled job that asks for the status of each service.
2. Action: restart exactly the service that is broken.
3. Reporting: send a Slack message so humans know what happened and can decide whether to investigate further.

It is "low-cost" because no new system is added: everything runs on the existing Jenkins, using the same SSH credential described in the article on Jenkins as the control plane, with no agent to install on each server.

2. Three types of self-healing jobs on Jenkins

Write the intervals with Jenkins' H/15 and H/10 notation: H hashes the job name into a fixed minute offset, so scan jobs do not all hit the servers at the same moment. Every scan job should enable disableConcurrentBuilds: if the previous run has not finished, the next one queues instead of overlapping. The systemctl is-active command returns a status string and a non-zero exit code when the service is not running, which makes it a good fit for scripts.

Scanning services on a minion

// excerpt from a service scan pipeline stage
def phpVersion = sh(
    script: "ssh ${SSH_OPTS} deploy@${minion} \"php -v | head -n1 | cut -d' ' -f2 | cut -d'.' -f1,2\"",
    returnStdout: true).trim()
def fpmService = "php${phpVersion}-fpm"

def nginxStatus = sh(
    script: "ssh ${SSH_OPTS} deploy@${minion} 'systemctl is-active nginx 2>/dev/null || echo inactive'",
    returnStdout: true).trim()

if (nginxStatus != 'active') {
    sh "ssh ${SSH_OPTS} deploy@${minion} 'sudo /usr/local/sbin/heal-service.sh nginx'"
}

Two details are worth noting. First, the PHP version is detected directly from php -v, so a single job can serve every minion group running PHP 7.2, 7.4 or 8.x without declaring a service name for each server (see the article on running multiple PHP versions). Note that plain php follows update-alternatives; if a server has several versions, list the php*-fpm units instead of probing the CLI. Second, || echo inactive turns a failure of the check command itself into a defined value, instead of turning the pipeline red.

Reading new log lines without rereading old ones

On each scan, the job takes the tail of the PHP-FPM and nginx logs and keeps only lines with a timestamp newer than the marker saved on the previous run (written to a small file on that same server), dropping notice-level lines. That way Slack only receives new errors instead of repeating an old alert every cycle. One limitation to be aware of: if the number of lines fetched is too small compared to the volume of log produced in one cycle, earlier lines are skipped.

Self-healing for supervisor: validate names before acting

The supervisor job usually extracts the domain name from the program name using the <domain>-worker convention. Before running php artisan queue:restart and supervisorctl start, it must check that the domain contains only letters, digits, dots and hyphens, and that the site directory exists. Any automatic fix based on a string taken from another server's output must always go through a check like this; otherwise an unexpected program name could become a command executed with high privileges.

3. Writing a safe self-healing script

Self-healing through a "bare" restart (no config check, no limit on attempts) is dangerous. The minimal skeleton for a self-healing script running on the target server:

#!/bin/bash
# /usr/local/sbin/heal-service.sh <service>: called from Jenkins over SSH
set -u
SERVICE=$1
STATE=/var/lib/heal/${SERVICE}.restarts
MAX_RESTARTS=3          # at most 3 times
WINDOW=3600             # within 1 hour
mkdir -p /var/lib/heal

state=$(systemctl is-active "$SERVICE" 2>/dev/null || true)
[ "$state" = "active" ] && exit 0                 # healthy, nothing to do

now=$(date +%s)
recent=$(awk -v n="$now" -v w="$WINDOW" 'n-$1<w' "$STATE" 2>/dev/null | wc -l)
if [ "$recent" -ge "$MAX_RESTARTS" ]; then
    echo "GIVE_UP: $SERVICE restarted $recent times in the window, needs a human"
    exit 2
fi

# nginx: check the config BEFORE restarting
if [ "$SERVICE" = "nginx" ] && ! nginx -t 2>/tmp/nginx-test.err; then
    echo "CONFIG_BROKEN: nginx -t failed, NOT restarting"; cat /tmp/nginx-test.err
    exit 3
fi

echo "$now" >> "$STATE"
systemctl restart "$SERVICE"
sleep 3
systemctl is-active --quiet "$SERVICE" && echo "RESTARTED: $SERVICE" || { echo "STILL_DOWN: $SERVICE"; exit 4; }

Each part of the script answers a specific question:

• nginx -t before restarting: if nginx died because of a bad config file, a restart will just fail again. According to the nginx control documentation, a syntax check is a mandatory step before any change.
• A limit on attempts: a service that restarts several times within an hour is not a random incident; it needs a human. The script must know when to "give up" and raise the alarm.
• Re-check after the restart: a successful systemctl restart does not mean the service is still alive a few seconds later. For PHP-FPM, also check the socket or send a request to a health check page.
• Distinct exit codes: so Jenkins can tell fixed, gave up and broken config apart, and pick the matching Slack color.

For supervisor, automatic restarts are configured right in the program file. According to the supervisor configuration documentation, autorestart and startretries determine how many times supervisor retries on its own before leaving the program in FATAL. The periodic scan job is only the second self-healing layer, catching the workers that supervisor has given up on.

4. The biggest risk of self-healing: hiding the root cause

Self-healing treats the symptom, not the cause. It can let a problem persist for a very long time without anyone noticing:

A classic example is a job that restarts PHP-FPM every few hours. When every site on a minion shares one pool, a heavy site can take all the workers and the other sites stall with it; a periodic restart wipes out the stuck workers and everything runs again. But that is only a temporary fix: the cause (a shared worker limit, overly slow requests) is still there. The real fix is to split PHP-FPM pools per site and track down the slow requests, which a later article covers.

A simple rule to avoid this trap: every self-healing action must leave evidence. The Slack message includes the server name, the service name, the state before and the result after. Restart counts over time must be visible, because "dozens of restarts a week" is very different from "one restart".

5. Limitations and a self-healing checklist

Limitations to be aware of in a periodic-scan self-healing model:

• The scan interval is the maximum downtime. Services that cannot tolerate that gap need an additional external check (uptime monitoring) or should let systemd restart them with Restart=on-failure.
• Jenkins is a single point of failure: if Jenkins dies, the self-healing layer goes with it. Alert when scan jobs have not run for a certain period.
• Servers are scanned sequentially: one server hanging on SSH slows down the whole scan. A short ConnectTimeout is mandatory.
• Static-threshold disk alerts do not measure how fast a disk fills up; a server that will be full in a few days is only reported on the last day.

Checklist for a self-healing job:

• Check the config before restarting; limit the number of attempts within a time window; re-check after the restart.
• Every string taken from another server is validated before it goes into a command.
• Every action sends a notification with full context; group them per scan run so Slack does not become noise.
• Count and graph self-healing actions per server and per service.
• Every repeated restart must lead to a root cause investigation.

Self-healing buys time, not answers. It follows the same "let the machines do the repetitive work" mindset as the nightly database backup article: the more you automate, the more you need signals telling you it is working correctly.

Next article

Demo domains and the lesson of a Google index leak. What happens when sites running on a demo domain get indexed by Google, and how to block it on the nginx side.