Hotfix Procedure
Emergency patch workflow, approval path, and post-incident steps.
Use this procedure when production is broken and a normal release cycle is too slow.
When to hotfix
- Error rate > 5% for 5+ minutes
- Critical user flow broken (login, checkout, data access)
- Security vulnerability actively exploited
Hotfix workflow
- Declare incident in #engineering and page on-call SRE.
- Branch from latest production tag:
hotfix/<issue-id>-<short-desc>. - Implement minimal fix — no unrelated changes.
- Get expedited review from code owner (target: 30 min).
- Deploy to staging, run smoke tests.
- Deploy to production with SRE approval.
- Verify metrics return to baseline.
- Merge hotfix branch back to
main.
Approval path
| Severity | Approvers required |
|---|---|
| P1 (outage) | Code owner + SRE lead |
| P2 (degraded) | Code owner |
| P3 (minor) | Normal PR process |
Post-incident
- Write incident summary within 24 hours
- Schedule blameless postmortem within 3 business days
- Create follow-up tickets for root cause fixes