Skip to main content

Runbook — Incident Response

Basic playbook for production incidents: detect → triage → mitigate → communicate → postmortem.

0. Severity Level​

LevelDefinitionResponse Time
SEV1All users unable to transact. Revenue impact.5 minutes
SEV2Some users affected / critical feature broken15 minutes
SEV3Degraded experience, workaround available1 hour
SEV4Internal/admin only, not user-facingNext business day

1. Detect​

Alert sources:

  • PagerDuty / Grafana alert
  • Sentry error spike
  • Customer support via #support-escalation
  • PSP / partner report

2. Triage — first 5 minutes​

Oncall must:

  1. Acknowledge the alert in PagerDuty
  2. Open #incident-active in Slack, post snapshot:
🚨 [SEV?] <short title>
Impact: <who is affected, how severe>
Detected: <time>
Status: investigating
  1. Ask "what changed in the last 1 hour?" — check:
    • Deploy log: ssh kesles-prod "journalctl -u dashboard-api --since '1 hour ago' | grep -i deploy"
    • Recent merge: git log --oneline --since='1 hour ago' main
    • Latest migration: check the most recent schema_migrations

3. Mitigate​

Rollback deploy (if suspect: recent deploy)​

→ see Deploy Dashboard API — Rollback

Rollback feature flag (if there is a flag)​

# via the admin dashboard or
curl -X POST https://kesles.com/merchant/dashboard-api/admin/flags/<flag-name> \
  -H "X-Internal-API-Key: ..." \
  -d '{"enabled": false}'

Scale horizontally (if load spike)​

ssh kesles-prod "systemctl start dashboard-api@worker{2,3,4}"

Disable the problematic endpoint (last resort)​

Edit reverse proxy config → return 503 + "scheduled maintenance".

4. Communicate​

  • Internal: update #incident-active every 15 minutes
  • Status page: https://status.kesles.com (update via admin)
  • Customer-facing: if SEV1/SEV2 > 15 minutes → support team sends broadcast email
  • Partner: if a partner API is affected → partner PIC is notified via the handover channel

5. Resolve​

Once the system has been stable ≥ 30 minutes:

✅ [SEV?] <title> — resolved
Root cause: <brief>
Duration: <start> → <end>
Impact: <actual>
Action items: <tracking>

6. Postmortem​

Mandatory for SEV1 + SEV2. Deadline 48 hours after resolve.

Template at merchant_docs/api_docs/internal/docs/dev/runbooks/postmortem-template.md (TBD — not yet created).

Minimum content:

  • Full timeline (detect, response, mitigation, resolve)
  • Root cause (5-whys)
  • Quantified impact
  • Action items with owner + due date
  • Blame-free analysis — focus on the system, not the person

Contacts​

  • Oncall rotation: see PagerDuty schedule kesles-backend-oncall
  • Tech lead: #eng-leads Slack
  • Security incident: directly to security@kesles.com + CTO