Lessons from a Late-Night Incident Response and What Changed After
A candid recap of root cause, fixes, and the new guardrails we shipped for resilience.
It was 11:47 PM. A message came in: "Site is down. Getting 502 everywhere."
This is a recap of that incident, what caused it, how we fixed it, and what we put in place so it never happens again.
The Context
A multi-tenant SaaS platform on a single Ubuntu 22.04 VPS (4 vCPU, 8GB RAM) running Nginx + PHP-FPM + PostgreSQL. About 2,000 active users across time zones.
Timeline
| Time | Event |
|---|---|
| 23:47 | First 502 reports from users |
| 23:49 | Me SSH'd in, checking nginx status |
| 23:52 | Identified PHP-FPM as the culprit |
| 00:08 | Service restored |
| 00:35 | Root cause confirmed |
Root Cause
A scheduled cron job ran a large database export at 23:30. It spawned 40+ PHP processes simultaneously, exhausting the pm.max_children limit in PHP-FPM. Nginx queued requests, then timed out. 502s everywhere.
# This showed the problem
journalctl -u php8.2-fpm --since "23:30" | grep "WARNING"
# WARNING: [pool www] server reached pm.max_children setting (10)
# WARNING: You may need to increase this value
Immediate Fix
# Bumped max_children temporarily
sed -i 's/pm.max_children = 10/pm.max_children = 25/' /etc/php/8.2/fpm/pool.d/www.conf
systemctl reload php8.2-fpm
Site came back within seconds.
Long-Term Guardrails
1. Rate-Limited the Cron Job
# Before (runs everything at once)
0 23 * * * php /var/www/app/artisan export:run
# After (batched with nice and ionice)
0 23 * * * nice -n 10 ionice -c 3 php /var/www/app/artisan export:run --batch-size=50
2. PHP-FPM Tuning
Set proper pool limits based on available RAM:
pm = dynamic
pm.max_children = 30
pm.start_servers = 5
pm.min_spare_servers = 3
pm.max_spare_servers = 10
pm.process_idle_timeout = 10s
pm.max_requests = 500
3. Added Monitoring Alert
# Grafana alert rule
condition: php_fpm_active_processes > 25
for: 2m
annotations:
summary: "PHP-FPM near capacity — check cron jobs"
What I Learned
Never run heavy jobs at peak-adjacent hours. And always have pm.status_path enabled so you can see FPM pool usage in real time.
A good incident response is 20% fixing the problem and 80% making sure it can't happen again.