The Production Operations series is a set of real incident reviews on Cloud VPS and Kubernetes, following symptom, investigation, root cause, fix and prevention: OOM, hanging PHP-FPM, slow MySQL, cache session leaks, WAF ...
If we had to fit every lesson from the three series (the three-tier Cloud VPS, Kubernetes for Website AI on Linode, and the operational incidents) onto a single page, we would choose an infrastructure...
A very common architecture as a system grows: the central application and satellite websites sit on the same private network, yet call each other's APIs through public domains, which means going out t...
This guide analyzes two typical incidents on two different infrastructures. On Cloud VPS, an internal SEO crawl made an e-commerce website return a flood of 500 errors, which in turn filled the memory...
A MySQL machine outside the Website AI Kubernetes cluster (a VPS on Linode) stopped responding. Prometheus recorded the exact moment, up went from 1 to 0, but no alert of any kind was sent, because Pr...
There is a kind of operational problem that is very hard to see: it runs silently, does not take down any website, but burns resources every day and buries the real signals. This guide covers two typi...
When a MySQL machine serving a Kubernetes cluster becomes too small for the amount of data it carries, the sensible plan is consolidating MySQL: retire that machine and move the websites using it to t...
An e-commerce website running Laravel Octane on a minion returns 502 because the kernel killed the site's FrankenPHP process after it ran out of RAM. One site running out of RAM is nothing unusual. Wh...
A Website AI site on Linode's LKE cluster returns 503 on and off. There is no long outage, just pages that occasionally fail to open and then work again a moment later. kubectl shows that the site's o...
A common situation on Website AI: an administrator cannot complete an image upload in the editor. The JPEG file is only a few MB but has a very high resolution (about 30 megapixels). Sometimes it fail...
A customer's mobile app could not update user profiles or delete transactions: every PUT and DELETE request received a 403. This is a textbook WAF false positive on a balancer running Coraza WAF. Whil...
An unpleasant situation on Cloud VPS: Grafana shows several machines at "CPU close to 100%" at the same time, or losing monitoring data entirely, including the balancer, the database machine and the m...
A fastcgi_cache configuration that works well on a pure content site is very easy to copy to another site. On an e-commerce site with a shopping cart, that same configuration can create two vulnerabil...
A very common misconfiguration on minions that run many sites: the config file declares fastcgi_cache_key and fastcgi_cache_valid, but has no fastcgi_cache_path and no fastcgi_cache directive. Page ca...
All articles are displayed
