Jenkins as the control plane for hundreds of websites

Once a cluster has dozens of Cloud VPS instances, the volume of repetitive daily operations is large: creating sites, deploying code, changing domains, running backups, removing sites, running an artisan command on the right minion. If each of these tasks is a script sitting on one person's machine, or a chain of SSH commands typed by hand, the outcome depends on that person's memory. This guide shows how to use Jenkins as a "control plane": a single place to trigger work, with parameters, schedules and logs, and the only place that holds the SSH keys to every server. It covers the core concepts, how to choose between freestyle and pipeline jobs, how to organize jobs into groups, parameter validation, risks, and a hardening checklist.

1. What Jenkins is

Jenkins is an open-source automation server. Its unit of work is a job: a set of commands plus configuration for parameters, schedules and notifications. Each run of a job is a build with a sequence number, its own log, a success or failure status, and a record of who triggered it. Commands execute on an agent (in a small cluster, the controller itself can act as the agent). Jobs run in three ways: triggered by hand, on a cron schedule, or through the API.

The strength of Jenkins lies in its plugin ecosystem. As a control plane for Cloud VPS, you only need a few plugins: SSH Agent to load a private key into a build, Credentials to store keys, Slack Notification to report results, and Pipeline to write multi-step jobs. The architecture overview is in the Jenkins User Handbook.

2. Freestyle or Pipeline

Jenkins has two common job types, and a control plane usually uses both:

A practical rule: if a job fits in about twenty lines of bash, keep it freestyle. If a job has three or more steps where each step depends on the result of the previous one, write it as a pipeline.

3. The control plane model: Jenkins SSHes into every server

                 operator (browser, over VPN)
                               │  Build with Parameters
                               ▼
   ┌─────────────────────────────────────────────────┐
   │ JENKINS (private network)                        │
   │  SSH credential ── ssh-agent plugin              │
   │  cron schedule ── parameters ── logs ── Slack    │
   └──────┬───────────────┬───────────────┬──────────┘
          │ ssh deploy@...│               │
          ▼               ▼               ▼
      balancer         minion        operations host ── mysql/mysqldump/rclone
                                            └──▶ database server

Jenkins does not need an agent installed on each Cloud VPS. It only needs to reach each server over SSH by its name in internal DNS (see the article on internal DNS). All MySQL work goes through the operations host, following the principle in the three-tier model article. A new server built with cloud images and cloud-init only needs the Jenkins public key added to come under the control plane.

For freestyle jobs, the key is loaded by the SSH Agent build wrapper. For pipelines, use the sshagent step:

pipeline {
    agent any
    options { disableConcurrentBuilds() }
    parameters {
        choice(name: 'minion', choices: ['app-php74-1', 'app-php82-1', 'app-php82-2'])
        string(name: 'domain', defaultValue: '', description: 'e.g. example.com')
    }
    stages {
        stage('Check Web Exists') {
            steps {
                sshagent(['ssh-ops-key']) {
                    sh '''
                      ssh -o ConnectTimeout=10 deploy@"$minion" \
                          "test -d /var/www/$domain" \
                        || { echo "/var/www/$domain not found on $minion"; exit 1; }
                    '''
                }
            }
        }
    }
    post {
        failure { slackSend(color: 'danger', message: "FAIL ${env.JOB_NAME} #${env.BUILD_NUMBER}: ${params.domain}") }
    }
}

The private key exists only in the Jenkins Credentials store. It is loaded into ssh-agent memory while the build runs, is never written to the workspace, and never shows up in the log.

4. Organizing Jenkins jobs by function

Name jobs with the convention <group>_<scope>, for example backup_app-php82-1 or site_create. The Jenkins search box then becomes an operations "menu": type backup_ and you see every backup job. You can group them further with folders or views per group.

Cron schedules in Jenkins should use the H symbol, for example H 19 * * *. Jenkins hashes the job name into a fixed minute within that hour, so dozens of backup jobs do not all hit the operations host at the same moment.

5. Parameterization: stopping typos at the input

The biggest advantage of a Jenkins job over a script is the parameter form:

• Choice for balancer and minion names: picked from a list, not typed, so nothing runs against a server that does not exist.
• String for domain, repo and branch. The value is always re-validated in the very first step.
• Boolean for options such as whether cron, supervisor or SSL issuance is needed.
• File for uploading an SQL file when creating a site from existing data. The job must check that the file has the right format before using it.

The domain goes straight into paths and shell commands, so it must be validated before anything else happens:

if echo "$domain" | grep -qE '\.\.|[;&|<>`$]'; then
    echo "Error: domain contains invalid characters: $domain"; exit 1
fi
if ! echo "$domain" | grep -qE '^[a-zA-Z0-9.-]+$'; then
    echo "Error: only letters, digits, dots and hyphens are allowed"; exit 1
fi
if ! ssh deploy@"$minion" "[ -d /var/www/$domain ]"; then
    echo "Error: /var/www/$domain does not exist on $minion"; exit 1
fi

These three checks stop both accidental mistakes (a stray space, the wrong minion) and dangerous strings such as ../ or ; rm -rf in the domain field.

6. Risks of using Jenkins as a control plane

Concentrating access in one place is convenient, but it also concentrates the risk in that place. Common risks and how to reduce them:

• Jobs not in Git: scripts written directly in the UI live in config.xml, so editing a job means editing production directly, with no review and no diff. Keep a Jenkinsfile in the repo and have the job read it from SCM, or at minimum export config.xml to Git on a schedule.
• One high-privilege key for every server: whoever can configure a job has power over the whole cluster. Use a deploy user with limited sudo instead of root, enforce strict Jenkins permissions (who may create, edit and run which jobs), and restrict the network so only Jenkins can SSH into servers.
• "Run any command" jobs: very handy for developers, but essentially a shell with a web UI. Limit who can run it, limit the allowed command set, and keep the build history (who ran what).
• Server lists scattered across jobs: adding or removing a minion means editing choice lists in many places, and missing one sends a job to a server that no longer exists. Move the server list into a shared environment variable or a config file in the repo.
• Jenkins as a single point of failure: if Jenkins dies, there are no backups and no deploys. JENKINS_HOME (jobs, credentials, build history) must be backed up like any other critical data.
• StrictHostKeyChecking=no: convenient when servers are rebuilt constantly, but it skips verifying the target server's identity. Manage known_hosts for Jenkins instead of disabling the check.
• Two builds touching the same site: disableConcurrentBuilds() only blocks within a single job. To block across different jobs (a domain change during a backup), use a per-site lock, for example the Lockable Resources plugin.

7. Jenkins control plane checklist

• Jenkins is reachable only over VPN or the private network, with authentication and role-based permissions enabled.
• Private keys live in Credentials and are loaded via SSH Agent; target servers use a dedicated deploy user.
• Every repetitive operation is a job with parameters, input validation, logs and notifications.
• Multi-step jobs are written as pipelines with post { failure } error notifications.
• Job scripts live in Git and are reviewed like application code.
• Cron uses the H symbol; jobs that write data take a per-site lock.
• JENKINS_HOME is backed up and a restore has been tested.

A control plane does not have to be complex: Jenkins, one SSH credential, internal DNS and a consistent directory convention are enough to drive dozens of Cloud VPS instances. Newcomers do not need to memorize commands, they only need to know which job to pick.

Next article

Nightly database backup: mysqldump, rclone and object storage. How the backup job on Jenkins works, how many days to keep, and the failures that silently break backups without anyone noticing.