The application reports SQLSTATE[HY000] [2002] Connection refused while the MySQL Service still has an address and answers ping, and the pod's CPU and RAM are completely normal. This is the typical shape of an incident when the MySQL volume is full: mysqld cannot start because the data disk has run out of space, so there is no socket to connect to. If you suspect a MySQL volume full condition, follow the steps below. This article follows the order of diagnosis and fixing: recognize the problem, find the culprit (binlogs), get MySQL running again, rebuild a larger volume, and then the pitfalls of resizing the PVC in place.

The article Volume, PersistentVolume, PVC and StorageClass explains the storage concepts used when a MySQL volume full condition strikes. If your MySQL runs as a StatefulSet, also see MySQL replication in a StatefulSet.

Symptom of a MySQL volume full condition: Connection refused

In a MySQL volume full incident, error 2002 is usually read as a wrong address or MySQL not having started. The first two tests are to check the network from the application pod to the Service, and to go straight into the MySQL pod and log in:

kubectl exec -it -n  db-1-0 -- bash
mysql -u root -p
# ERROR 2002 (HY000): Can't connect to local MySQL server
#   through socket '/var/lib/mysql/mysql.sock' (2)

A healthy network with a missing socket means mysqld is not running, and a MySQL volume full condition is the prime suspect. Grafana shows normal CPU and RAM, so the investigation has to turn to the disk.

Diagnosing a MySQL volume full condition with df and du

df -h /var/lib/mysql
# Filesystem   Size  Used Avail Use% Mounted on
# /dev/...     9.8G  9.8G     0 100% /var/lib/mysql

du -hsx /var/lib/mysql/* | sort -h | tail
# 1.1G  /var/lib/mysql/binlog.000001
# 1.1G  /var/lib/mysql/binlog.000002
# ...

The 100% line confirms the data volume is full. The du command shows that most of the space belongs to a series of binlog.NNNNNN files, about 1 GB each, while the real data takes only a small part. Note that du -x stays within one filesystem, so it does not miscount parts of the overlay or other mounts.

Why binlogs make the MySQL volume full

The binary log records every data change and is needed for replication and point-in-time recovery. Since MySQL 8.0 the binary log is enabled by default, while older images often have it off. max_binlog_size (default 1 GB) only decides the size of each file; a very large transaction can still push a file past this value.

Retention is decided by binlog_expire_logs_seconds (default 2592000 seconds, which is 30 days), so the limit is by age and not by file count. The total binlog size depends on the write volume multiplied by the retention time, so with a 10 GB volume and a heavy write burst such as a bulk product import, the volume fills very quickly, and keeping 30 days is far too long. The reason binlogs grew so fast in one specific incident has not been proven; it is usually only suspected from a large import. The two defaults above come from the MySQL documentation, and you should compare them with the version you are running.

Getting MySQL to start when the disk is out of space

With a MySQL volume full condition, when mysqld cannot start you cannot run PURGE because there is no connection. The procedure has two steps: free the minimum space needed for mysqld to start, then clean up properly with MySQL's own command.

Step 1, free some space. There are three options, in order of preference:

  • Resize the PVC first if the StorageClass allows it (see the PVC resize section below).
  • Scale the StatefulSet to 0, mount the PVC into a temporary pod, delete the oldest binlog file and edit binlog.index to match (remove the line of the deleted file), then scale back up.
  • The most makeshift way: mv a binlog file out of the volume. That file sits on the container's overlay and is lost when the container restarts; binlog.index is not updated, so MySQL still believes the file exists. Use it only when there is no other way.

Step 2, clean up properly once mysqld is running:

-- after mysqld is running again
PURGE BINARY LOGS TO 'binlog.000006';
-- or by time
PURGE BINARY LOGS BEFORE NOW() - INTERVAL 1 DAY;

If there are replicas, check the replica's read position with SHOW REPLICA STATUS before PURGE, because deleting binlogs a replica has not read yet breaks replication. Try this procedure in a test environment first.

Before any deletion, dump the data and verify the dump. Use the password from an environment variable already in the pod (taken from a Secret) instead of typing it on the command line:

kubectl exec -i -n <namespace> db-1-0 -- sh -c \
  'MYSQL_PWD="$MYSQL_ROOT_PASSWORD" mysqldump -u root --single-transaction \
   --routines --triggers --events app_db' > app_db.sql
ls -lh app_db.sql
tail -n 3 app_db.sql   # you must see the "Dump completed" line

The variable $MYSQL_ROOT_PASSWORD is inside single quotes, so the shell in the pod expands it and the password never appears in the arguments of kubectl. If your image names the variable differently, use that name, or use --defaults-extra-file mounted from a Secret. If the last line of the dump file is not -- Dump completed, the dump is broken. Deleting the PVC before the dump is verified risks real data loss, especially when the MySQL volume is full and you are in a hurry. Do not type the password directly on the command line: it shows up in shell history and in the process list.

Rebuilding a larger volume after a MySQL volume full incident

One reliable fix for a MySQL volume full condition is to rebuild from scratch: dump, delete the StatefulSet, delete the PVC, delete the PV, increase the size, apply, restore. The steps in order:

  1. Dump and verify the dump as above.
  2. Delete the StatefulSet, then the PVC.
  3. Look at the PV: if the StorageClass has reclaimPolicy: Retain, the PV moves to the Released state (this is a PV state, not a PVC state).
  4. Delete the PV. This only deletes the object in the cluster; with Retain the disk at the provider remains and is still billed, so it needs to be cleaned up separately.
  5. Edit volumeClaimTemplates with the new size, apply the StatefulSet again, then restore the dump.
volumeClaimTemplates:
  - metadata:
      name: data
    spec:
      accessModes: ["ReadWriteOnce"]
      resources:
        requests:
          storage: 30Gi
      storageClassName: block-storage-retain   # example name

The StorageClass name above is only an example; on LKE, see the real name with kubectl get storageclass.

When the MySQL volume is full, this is the most reliable method but it involves downtime, and you must be sure the data is safely stored outside the PVC.

StatefulSet limits when a MySQL volume full condition needs more space

Trying to edit storage in the volumeClaimTemplates of a running StatefulSet gives the error Forbidden: updates to statefulset spec for fields other than 'replicas', 'ordinals', 'template', 'updateStrategy'... are forbidden. This is a general StatefulSet limitation (see how to build one in your first Kubernetes manifest), not something new. The alternative is to resize the PVC directly, but the StorageClass needs allowVolumeExpansion: true:

kubectl get storageclass -o custom-columns=NAME:.metadata.name,EXPAND:.allowVolumeExpansion
kubectl patch pvc data-db-1-0 -n  -p '{"spec":{"resources":{"requests":{"storage":"20Gi"}}}}'

Use kubectl patch or kubectl edit on the PVC object itself. A common mistake is to change the size by pasting a PersistentVolumeClaim manifest into the StatefulSet file and applying it; that does not resize an existing PVC.

Part two: resizing the PVC for a MySQL volume full condition does not go as expected

This part describes one attempt to resize a PVC on a volume that had been filled on purpose by copying a large file. After editing the PVC to 20Gi, df -h in the pod still reported about 9.8G. Resizing a PVC has two phases: phase one expands the volume on the provider side, and phase two expands the filesystem, which the CSI driver does when the volume is mounted. So check the PVC conditions instead of guessing:

kubectl describe pvc data-db-1-0 -n <namespace>
# Conditions:
#   FileSystemResizePending   True
kubectl get pvc data-db-1-0 -n <namespace> -o jsonpath='{.status.capacity.storage}'
kubectl get volumeattachment

The FileSystemResizePending condition means phase two is waiting. In describe pvc, also look at the Resizing and ExternalExpanding events to see which phase is slow, while get volumeattachment shows which node the volume is attached to. Running resize2fs directly in the container is a wrong inference: the tool is not in the image and the job belongs to the CSI driver, not the application.

In that attempt, deleting the pod to force a remount led to a long ContainerCreating with the event volume attachment is being deleted; the pod took quite a while to come up and df still showed the old size, and restarting the node did not change it either. That is only the result of one attempt with one provider, it should not be taken as common behavior, and the final outcome was not confirmed.

The command that raised the PVC size was accepted by the API server; if the StorageClass did not have allowVolumeExpansion enabled it would normally be rejected right away, so you can infer that the flag was on, but you should still verify with the kubectl get storageclass command above. For similar volume attachment errors, see Pod stuck in ContainerCreating on LKE.

Preventing a MySQL volume full condition

  • To avoid a MySQL volume full condition, set a suitable binlog_expire_logs_seconds: this parameter decides the retention time, while max_binlog_size only decides the size of each file (lowering it allows finer cleanup but does not reduce the total size). For example in my.cnf: binlog_expire_logs_seconds=259200 (3 days). This is an example value, not tried on a cluster; check it before applying.
  • Turn off the binary log if you use neither replication nor point-in-time recovery (skip-log-bin).
  • Alert early with Prometheus (see Kubernetes monitoring with kube-prometheus), for example when free space drops below 20%: kubelet_volume_stats_available_bytes / kubelet_volume_stats_capacity_bytes < 0.2.
  • Choose a StorageClass with allowVolumeExpansion: true from the start so that a MySQL volume full condition still has a quick way to grow.
  • Rehearse the restore procedure regularly, because a dump that has not been verified is not yet a backup.

Checklist for a MySQL volume full condition

  • Suspect a MySQL volume full condition when you see Connection refused with normal CPU and RAM: run df -h on the data volume first.
  • Once mysqld is up, use PURGE BINARY LOGS (check replicas first); use a temporary mv only when there is no other way.
  • For any MySQL volume full recovery, verify the dump before deleting the StatefulSet, PVC or PV.
  • After deleting a Retain PV, clean up the disk at the provider to avoid being billed.
  • After fixing the MySQL volume full condition, set a capacity alert so you find out early next time.
  • When resizing a PVC, check describe pvc and the FileSystemResizePending condition.

Next article

Installing mysqld_exporter on Kubernetes: three silent errors that leave Prometheus empty. Volume capacity alerts come from the kubelet, while this next article goes on to the metrics inside MySQL: how to install mysqld_exporter and the three errors that leave Prometheus with no data.