Incident Runbook
Sample template — severity levels, response steps, and comms.
Service: service-name · On-call: rotation / PagerDuty schedule · Dashboards: link
Severity levels
| Sev | Impact | Response time | Example |
|---|---|---|---|
| SEV1 | Full outage / data loss | Immediate, all-hands | API down, checkout broken |
| SEV2 | Major degradation | < 30 min | High error rate, key feature down |
| SEV3 | Minor / partial | < 1 business day | Slow endpoint, cosmetic bug |
Response loop
First 15 minutes
- Acknowledge the alert so the team knows it's being handled.
- Declare severity and open an incident channel — #inc-yyyy-mm-dd-short-name.
- Assign roles: Incident Lead (coordinates), Comms (updates stakeholders).
Mitigate first: roll back the last deploy (Rollback), disable the feature flag, or scale up — before root-causing.
- Post status every 15 min, even if the update is "still investigating".
Common mitigations
| Symptom | First action |
|---|---|
| Spiking errors after a deploy | Roll back to the previous release |
| One bad feature | Toggle its feature flag off |
| Resource exhaustion | Scale out ECS tasks / increase limits |
| Dependency down | Enable degraded mode / circuit breaker |
See Common Issues for service-specific diagnostics.
Communication
- Internal: incident channel + #status updates every 15 min
- External: update the status page for SEV1/SEV2 ( link )
- Resolution: post an all-clear and confirm metrics are back to baseline
After resolution
- Confirm error rate / latency back to baseline for 30 min
- Write a blameless postmortem within 48h (timeline, root cause, action items)
- File follow-up tickets for every action item with an owner and due date