Production Rollback

Safe rollback steps, verification, and communication checklist.

Rollback triggers

Error rate exceeds 5% for 5 minutes
P95 latency doubles from baseline
Critical user flow broken
Data integrity issue detected

Kubernetes rollback

bash

kubectl -n production rollout history deployment/backend-api
kubectl -n production rollout undo deployment/backend-api
kubectl -n production rollout status deployment/backend-api

Vercel rollback

bash

vercel rollback [deployment-url]

Or via dashboard: Deployments → select previous → Promote to Production.

Communication checklist

Post in #engineering: what's broken, which deployment, rollback in progress.
Page on-call SRE if P1.
Execute rollback command.
Verify health endpoint and error rate return to baseline.
Update #engineering with resolution status.
Schedule postmortem within 3 business days.