There is a kind of operational problem that is very hard to see: it runs silently, does not take down any website, but burns resources every day and buries the real signals. This guide covers two typical cases from the same family. First, cron jobs on Jenkins still SSH into machines that were removed long ago, failing constantly without anyone noticing. Second, the MySQL error.log on a database machine of the Kubernetes cluster grows to several GB, almost all of it a single reverse DNS warning repeated millions of times. The article covers how to detect them, the root causes, the configuration fixes (skip_name_resolve, logrotate) and a checklist for removing a machine.

Signs

No customer reports an error. The signs only show up when you actively look:

  • A cron job on Jenkins (for example clearing the file cache of a few Laravel sites, running at several times a day) has mostly FAILURE results among its recent builds, and has not had a single successful build for a long time.
  • A job with a required parameter but a periodic trigger fails like clockwork every evening.
  • An SSL renewal job for a removed balancer still runs every week and fails every time.
  • On the MySQL machine outside the cluster (the model described in MySQL in Kubernetes or outside the cluster: why we moved the database to a VPS), the MySQL error.log takes up several GB in /var/log.

Investigating zombie cron jobs

Step 1: count builds per month instead of looking at the latest build

The Jenkins UI only shows the last few builds. To find out when a job started failing, read each build's build.xml in JENKINS_HOME directly (read-only) and group by month:

cd $JENKINS_HOME/jobs/clear-cache-job/builds
for b in [0-9]*/build.xml; do
  m=$(grep -m1 -o '<timestamp>[0-9]*' $b | cut -c12-21)
  r=$(grep -m1 -o '<result>[A-Z]*' $b | cut -c9-)
  echo "$(date -d @$m +%Y-%m) $r"
done | sort | uniq -c

A common pattern: the job fails occasionally, then fails more often than it succeeds, then fails completely. People are so used to seeing red that nobody notices the difference when it breaks for good.

Step 2: read the console log, and one wrong lead

The console usually shows two kinds of errors interleaved:

rm: cannot remove 'storage/framework/cache/data/': Directory not empty
ssh: Could not resolve hostname minion-old: Name or service not known

It is easy to think the main error is the rm line: the rm -rf command runs at the exact moment the application writes new cache entries, so the directory occasionally cannot be removed. That error is real, but it only explains the occasional failures. When every build stops at the second line, the cause is a script with a hardcoded machine name:

ssh root@minion-old "cd /var/www/site-a.example && rm -rf storage/framework/cache/data/ && chown -R www-data:www-data *"
ssh root@minion-old "cd /var/www/site-b.example && rm -rf storage/framework/cache/data/ && chown -R www-data:www-data *"

That machine had been slowly failing (the slow death described in no live upstreams: reading the Nginx error.log when upstreams are marked down), its sites were moved away, the machine was removed and its name was deleted from internal DNS. Nobody fixed the job. Probe ports 22 and 80 from another machine to confirm that the machine is gone.

Digging further through scheduled jobs usually turns up more:

  • An SSL renewal job targeting a removed balancer by name, failing with Could not resolve hostname.
  • Jobs with required parameters (minion, domain) but a periodic trigger. Jenkins cron runs them with the default value, which is empty, so they stop immediately at the check [ERROR] domain parameter is required. The minion choice list still includes removed machines.
  • "Zombie rules" in Grafana: CPU and RAM rules pinned to the instance of a removed machine, set to No data → OK, which means that since the machine disappeared, the rule reports "normal" forever.

Investigating the bloated MySQL error.log

Step 3: what is in the MySQL error.log

If there is no logrotate configuration for MySQL (no MySQL file in /etc/logrotate.d), the MySQL error.log is a single file that has been growing since the last boot. Count by warning code:

ls -la /var/log/mysql/
grep -oE '\[MY-[0-9]+\]' /var/log/mysql/error.log | sort | uniq -c | sort -rn | head
grep MY-010055 /var/log/mysql/error.log | head -1
[Warning] [MY-010055] [Server] IP address '10.0.1.13' could not be resolved: Name or service not known

When one code takes up almost the whole file, group it by IP over a sample day:

grep MY-010055 error.log | grep "$(date -d yesterday +%F)" \
  | grep -oE "IP address .[0-9.]+" | sort | uniq -c | sort -rn

In the typical case, the top IPs are the worker nodes of the LKE cluster. Website pods connect from the nodes, the nodes' private IPs have no PTR record on the database machine's DNS resolver, so every time MySQL asks for a name it gets "Name or service not known" and writes a warning line. With hundreds of pods connecting continuously, tens of thousands of lines are added every hour.

The rest are usually unknown IPs from the Internet: public port scanning services. Seeing them in the MySQL error.log means port 3306 is answering from the Internet and must be closed right away.

Step 4: real signals buried in the MySQL error.log

With MY-010055 filtered out, the MySQL error.log often still contains a few things worth reading that nobody had noticed before, for example Background histogram update on <db>.<table>: Lock wait timeout exceeded repeated hundreds of times. Also check table_open_cache:

SHOW GLOBAL STATUS LIKE 'Table_open_cache_overflows';
SELECT @@table_open_cache, COUNT(*) AS table_count FROM information_schema.tables;

Overflows growing quickly while the total number of tables across all databases exceeds table_open_cache is a sign that this value should be raised.

Root causes: zombie cron jobs and the MySQL error.log

Zombie cron jobs

  • Removing a machine has no "find everything that points to this machine" step. DNS was cleaned up, Prometheus targets were cleaned up, but Jenkins jobs and Grafana rules were not. The jobs hardcode hostnames instead of reading from a shared machine list.
  • Job configuration that is not in Git cannot be searched with one grep across all jobs the way you grep a repo (see Jenkins as the control plane for hundreds of websites).
  • Jobs with required parameters were given a cron trigger.
  • Nobody receives an alert when a job fails repeatedly. Red becomes the background, and nobody looks at the background.

The bloated MySQL error.log

  • The default is skip_name_resolve = OFF. For each new client host, MySQL resolves the IP to a name and the name back to an IP to compare with the account's host column, and caches the result in the host cache. Private IPs have no PTR record, so every lookup fails and produces a warning line. If every account is defined by IP or %, this name resolution brings no benefit at all.
  • There is no logrotate for the MySQL error.log, so a harmless warning piles up into a multi-GB file and hides the real errors.
  • Port 3306 open to the Internet lets scans pump even more lines into the log and widens the attack surface for no reason.

Remediation

Clean up the zombie cron jobs

  1. Disable the jobs pointing at removed machines. Disable, do not delete yet, so the build history is still available for comparison.
  2. Remove the periodic trigger from jobs with required parameters; remove the retired machines from the choice lists.
  3. Delete Grafana rules pinned to removed machines.

Fix the MySQL error.log

Archive the current MySQL error.log somewhere else, then make MySQL open a new file: rename the old file, run FLUSH ERROR LOGS, compress the old file.

Enable skip_name_resolve, but only after making sure no account is defined by hostname, because once it is enabled, those accounts can no longer log in:

-- run before enabling: must return nothing (apart from localhost)
SELECT user, host FROM mysql.user
WHERE host NOT IN ('%', 'localhost')
  AND host NOT REGEXP '^[0-9.%/]+$';

# /etc/mysql/mysql.conf.d/mysqld.cnf
[mysqld]
skip_name_resolve = ON

Add logrotate for the MySQL error.log, using mysqladmin flush-logs error in postrotate so MySQL writes to the new file:

# /etc/logrotate.d/mysql-error
/var/log/mysql/error.log {
    daily
    rotate 14
    maxsize 200M
    compress
    delaycompress
    missingok
    notifempty
    create 640 mysql adm
    postrotate
        mysqladmin --defaults-file=/root/.my.cnf flush-logs error
    endscript
}

Close port 3306 to the Internet for a cleaner MySQL error.log

Restrict port 3306 to the cluster's private network only, using the provider's firewall (for LKE, Linode Cloud Firewall) or nftables on the host, and set bind-address to the private IP if possible. Test from a machine outside the network: nc -zv <public-ip> 3306 must fail to connect. This change and raising table_open_cache need a maintenance window because they affect running connections.

Checklist for removing a machine

# find every Jenkins job that still mentions the machine being removed
grep -l "minion-old" $JENKINS_HOME/jobs/*/config.xml

# Prometheus targets, /etc/hosts on other machines
grep -rn "minion-old" /etc/prometheus/ /etc/hosts
  • Removing a machine is a change with a checklist: DNS, Prometheus targets, alert rules, Jenkins jobs (including the choice lists in parameters), backups, notification channels.
  • Periodic jobs do not hardcode machine names; they read from a shared machine list, and a machine removed from the list is skipped automatically.
  • Jobs with required parameters must not have a cron trigger. If they need to run periodically, split them into a separate job with fixed parameters. For the schedule syntax, see the Jenkins cron syntax documentation.
  • Alert when a cron job fails three times in a row, not on every failure.
  • Alert rules treat No data as an alerting state, not OK.

Monitoring the MySQL error.log

  • Every database machine has logrotate for the MySQL error.log and an alert when /var/log crosses a threshold.
  • Every month, count the top 10 warning codes in the MySQL error.log. Any code taking up more than half the file is a task to handle, not a line to ignore.
  • Record every change to mysqld.cnf (ideally manage it in Git), so that when MySQL's behavior changes after a restart, there is an old version to compare against.

Lessons learned

  • Familiar red is more dangerous than sudden red. Monthly statistics reveal what the latest-build screen hides.
  • Removing a machine is a configuration change in many places, not just clicking delete on a VPS.
  • A harmless warning is still harmful when it fills almost the entire log. A bloated MySQL error.log does not bring the database down, but it makes sure nobody reads the log, and the real errors sit quietly inside it.
  • No configuration change is "small" if it is not recorded, including a single line in mysqld.cnf that changes how the MySQL error.log behaves.

Next article

A database died and nobody knew: gaps in Prometheus alerting. When a MySQL machine stops responding while the monitoring system stays silent, and the minimum rule set to keep it from happening again.