What I Built Today: Staging Pipeline Upgrades and a Faster Deploy Loop

Quick daily log from today's work — mostly infrastructure plumbing but the kind that pays dividends for months.

What I Was Working On

A client's staging environment had a fragile deploy loop. The CI job would push code, SSH into the server, and run a flat bash script. No checks, no rollbacks, no notifications. Just hope.

Changes Shipped

1. Pre-flight Hook in CI

Added a pre-deploy.sh that runs before every deployment:

#!/bin/bash
set -e

echo "Running pre-flight checks..."

# Check disk space
DISK_FREE=$(df -h / | awk 'NR==2 {print $5}' | tr -d '%')
if [ "$DISK_FREE" -gt 85 ]; then
  echo "ERROR: Disk usage at ${DISK_FREE}%. Aborting deploy." >&2
  exit 1
fi

# Check service health
if ! systemctl is-active --quiet nginx; then
  echo "ERROR: Nginx is not running. Aborting deploy." >&2
  exit 1
fi

echo "Pre-flight checks passed."

2. Canary Deploy Step

Instead of hot-swapping the entire app, we now deploy to a canary directory first:

DEPLOY_DIR="/var/www/app"
CANARY_DIR="/var/www/app-canary"

rsync -az --delete ./dist/ "$CANARY_DIR/"
# Run smoke tests against canary
curl -f http://localhost:8080/health || { echo "Canary failed"; exit 1; }
# Swap
rsync -az --delete "$CANARY_DIR/" "$DEPLOY_DIR/"

3. Rollback in 30 Seconds

Kept the last 3 releases in /var/www/releases/ and updated the symlink on swap:

ln -sfn /var/www/releases/$(date +%Y%m%d%H%M%S) /var/www/current

Rollback is just pointing the symlink back.

What Improved

• Deploy time: 7 min → 2.5 min (pre-flight runs in parallel)
• Last 3 deploys survived without intervention
• Client gets a Slack notification on success or failure

Tomorrow

Setting up log-based alerting so we know within 60 seconds if a deploy causes error rate spikes.