Skip to main content

Runbook — Secret Rotation

Procedure to rotate the HMAC secret (PSP) and internal API key without downtime.

When To Rotate​

  • Scheduled: every 90 days (automated reminder in the admin dashboard)
  • Incident: when leak is suspected (log leak, employee turnover, breach)
  • Request: partner / PSP requests it via the support channel

HMAC Secret Rotation (PSP)​

Step 1 — Generate new secret​

# On a secure workstation, not the production server
openssl rand -base64 32
# Example output: "nst1aCm9...<44 char>"

Step 2 — Encrypt with the master key​

KESLES_SECRET_ENCRYPTION_KEY="<base64-master-key>" \
  go run ./services/dashboard_api/cmd/psp-encrypt "<new-secret>"
# Output: enc:v1:<nonce>:<ciphertext>

Step 3 — INSERT a new row into psp.api_keys​

INSERT INTO psp.api_keys (key_id, hmac_secret_encrypted, is_active, created_at)
VALUES ('psp-prod-key-2026-04', '<enc:v1:...>', true, now());

Important: leave the old row with is_active = true. Dual-key grace period of 7 days.

Step 4 — Deliver the new secret to the PSP​

Use an encrypted channel (never via email/Slack). Preferred: Bitwarden Send / one-time secret link.

Information delivered:

  • key_id (public, safe)
  • secret plaintext (shown once, not stored)
  • Grace period: start date + end date

Step 5 — Monitor adoption​

Query the psp.event_log:

SELECT key_id, COUNT(*) FROM psp.event_log
WHERE created_at > now() - interval '7 days'
GROUP BY key_id;

Once the PSP is 100% on the new key → revoke the old one.

Step 6 — Revoke old key​

UPDATE psp.api_keys
SET is_active = false, revoked_at = now()
WHERE key_id = '<old-key>';

Internal API Key Rotation​

Same flow, but in the internal.api_keys table. Endpoints that use it: /internal/*.

Master Key Rotation (KESLES_SECRET_ENCRYPTION_KEY)​

More complex because every row in psp.api_keys.hmac_secret_encrypted + ref_payment_service_provider.{api_key,webhook_secret}_encrypted must be re-encrypted before cut-over. The full procedure (pre-flight check, dry-run, commit, rollback, checklist) is in a separate runbook:

➡️ AES Master Key Rotation

Post-Rotation Checklist​

  • New row inserted + is_active = true
  • Secret delivered to the partner via secure channel
  • Grace period documented in the ops/ folder
  • Monitor logs for 7 days
  • Revoke old key (set is_active = false)
  • Update internal password manager (Bitwarden vault kesles-merchant-secrets)