Skip to main content

Runbook — AES Master Key Rotation (KESLES_SECRET_ENCRYPTION_KEY)

Procedure to rotate the AES-256-GCM master key used by dashboard-api to encrypt PSP/acquirer secrets in Postgres.

⚠️ High risk. Wrong order = all secrets in the DB cannot be decrypted → service down + must restore from backup. Read the entire runbook before starting.

Context​

What the master key protects:

TableColumnPlaintext value
psp.api_keyshmac_secret_encryptedHMAC secret to verify inbound requests from payment.kesles.com
ref_payment_service_providerapi_key_encryptedOutbound API key to the acquirer (BCA, Mandiri, etc.)
ref_payment_service_providerwebhook_secret_encryptedHMAC secret to verify callbacks from the bank

Ciphertext format in the DB: enc:v1:<base64-nonce>:<base64-ciphertext> — see services/dashboard_api/internal/app/psp_crypto.go.

Current env implementation (psp_crypto.go:50-63): reads only one env var KESLES_SECRET_ENCRYPTION_KEY. There is no _OLD fallback in the normal code path — re-encrypt must be performed by a separate script that explicitly reads both envs.

When To Rotate​

  • Safe to skip if:
    • Not yet real production (dev/staging with empty DB).
    • psp.api_keys + ref_payment_service_provider are still empty (zero rows with the enc:v1:* prefix).
  • Must rotate if:
    • Suspected leak (key exposed in repo, log, dump file, ex-employee).
    • Compliance schedule (e.g. annual key rotation policy).
    • Migration from a dev key (shared with the team) to a prod key (restricted access).

Pre-Flight Check​

Before doing anything, verify the DB state:

-- Connect to db_reference (or db_kesles_merchant depending on the deployment)
SELECT
  (SELECT count(*) FROM psp.api_keys
     WHERE hmac_secret_encrypted LIKE 'enc:v1:%') AS psp_keys_encrypted,
  (SELECT count(*) FROM ref_payment_service_provider
     WHERE api_key_encrypted LIKE 'enc:v1:%') AS acq_api_keys,
  (SELECT count(*) FROM ref_payment_service_provider
     WHERE webhook_secret_encrypted LIKE 'enc:v1:%') AS acq_webhook_secrets;

Scenario A — Greenfield (no encrypted data yet)​

No need for the 4-step procedure. Just:

  1. Generate a new key in the dashboard menu Settings → Secrets → AES-256 Master Encryption Key card → click Generate → copy. (Or via CLI: openssl rand -base64 32.)
  2. Edit services/dashboard_api/.env.production (on the prod server, not on your workstation): 
    KESLES_SECRET_ENCRYPTION_KEY=<new-base64-44-char>
  3. Restart dashboard-api: 
    sudo systemctl restart merchant-dashboard-api
    journalctl -u merchant-dashboard-api -n 50 -f
  4. Smoke test: open the PSP Endpoint Tester in the dashboard → send 1 GET request → expect status 200. If decrypt fails → key is wrong, rollback.

Done. The database is not touched.

Scenario B — Re-encrypt (existing encrypted data)​

Four official steps. Total downtime: ~5 minutes (restart window).

Step 1 — Back up old key + DB​

# 1a. Note the OLD_KEY value from the current prod env (do NOT share in a
#     public chat — use a password manager).
OLD_KEY=$(grep ^KESLES_SECRET_ENCRYPTION_KEY= /etc/kesles/.env.production | cut -d= -f2-)

# 1b. Snapshot the DB before touching anything.
sudo bash /opt/kesles/scripts/db_backup.sh --tag pre-master-key-rotation

Verify the backup was created in /var/backups/kesles/ and size > 0.

Step 2 — Generate the new key + add to env as a dual-key​

Generate via the dashboard Secrets menu or the CLI:

NEW_KEY=$(openssl rand -base64 32)
echo "$NEW_KEY"        # 44 char base64

Edit services/dashboard_api/.env.production on the prod server so that both keys are available (the primary key is still the old one, the new key is in the temporary _NEW slot — service still runs normally with the old key):

# Before:
# KESLES_SECRET_ENCRYPTION_KEY=<OLD_KEY>

# After:
KESLES_SECRET_ENCRYPTION_KEY=<OLD_KEY>           # not yet changed
KESLES_SECRET_ENCRYPTION_KEY_OLD=<OLD_KEY>       # alias for the script
KESLES_SECRET_ENCRYPTION_KEY_NEW=<NEW_KEY>       # rotation target

The names _OLD and _NEW here are read by the manual re-encrypt script, not by the service. The service still uses KESLES_SECRET_ENCRYPTION_KEY (which is still <OLD_KEY>).

No restart needed at this step.

Step 3 — Re-encrypt all rows​

Implementation status​

The re-encrypt CLI is not yet in the repo as of the current release. Create a one-shot Go script or run inline. The architectural skeleton agreed:

services/dashboard_api/cmd/psp-rotate-key/main.go

How it must work:

  1. Read KESLES_SECRET_ENCRYPTION_KEY_OLD + KESLES_SECRET_ENCRYPTION_KEY_NEW from env. Validate both as 32-byte base64.
  2. Connect to DB using the existing DATABASE_URL/DB_REFERENCE_URL.
  3. For each table + column (3 pairs):
    • SELECT id, <col> FROM <tbl> WHERE <col> LIKE 'enc:v1:%'
    • For each row:
      • plain := decryptAES(value, keyOld) — if it fails → log + skip.
      • newCipher := encryptAES(plain, keyNew)
      • Buffer to a list of (id, newCipher) first.
  4. --dry-run mode: print summary count old / count new / sample first 3 then exit.
  5. --commit mode: wrap each table in a transaction, run UPDATE per row, commit, log count.
  6. Verification post-commit: SELECT count(*) WHERE <col> LIKE 'enc:v1:%' must equal the count before.

Execution​

# Dry-run first — does not touch the DB
KESLES_SECRET_ENCRYPTION_KEY_OLD=<OLD_KEY> \
KESLES_SECRET_ENCRYPTION_KEY_NEW=<NEW_KEY> \
DATABASE_URL=postgresql://... \
  go run ./services/dashboard_api/cmd/psp-rotate-key --dry-run

# Expected output:
#   psp.api_keys.hmac_secret_encrypted              : 5 rows
#   ref_payment_service_provider.api_key_encrypted  : 12 rows
#   ref_payment_service_provider.webhook_secret_*   : 12 rows
#   total: 29 rows would be re-encrypted

# If the numbers match the pre-flight check → commit
KESLES_SECRET_ENCRYPTION_KEY_OLD=<OLD_KEY> \
KESLES_SECRET_ENCRYPTION_KEY_NEW=<NEW_KEY> \
DATABASE_URL=postgresql://... \
  go run ./services/dashboard_api/cmd/psp-rotate-key --commit

Verify:

-- After commit, the DB still has the same number of encrypted rows
SELECT count(*) FROM psp.api_keys WHERE hmac_secret_encrypted LIKE 'enc:v1:%';

Step 4 — Cut over the service to the new key, then drop the old key​

# 4a. Edit .env.production: move NEW_KEY to the primary slot.
KESLES_SECRET_ENCRYPTION_KEY=<NEW_KEY>            # ← cut over here
KESLES_SECRET_ENCRYPTION_KEY_OLD=<OLD_KEY>        # still kept for 24-48 hours
# KESLES_SECRET_ENCRYPTION_KEY_NEW removed

# 4b. Restart the service.
sudo systemctl restart merchant-dashboard-api

# 4c. Smoke test same as Scenario A:
#       open the PSP Endpoint Tester → send a request → expect 200.
#       Tail logs for "gcm decrypt: message authentication failed" errors.
journalctl -u merchant-dashboard-api -n 200 | grep -Ei 'decrypt|aes|master.key'

After 24-48 hours of smooth prod traffic (no decrypt errors at all, inbound + outbound requests 200 normal):

# Final cleanup — remove OLD_KEY from env.
sed -i '/^KESLES_SECRET_ENCRYPTION_KEY_OLD=/d' /etc/kesles/.env.production
sudo systemctl restart merchant-dashboard-api

Done. The old key is gone from env, memory, and dumps.

Rollback​

If there are decrypt errors after Step 4:

  1. Stop traffic to prod (load balancer / Caddy).
  2. Edit .env.production: revert KESLES_SECRET_ENCRYPTION_KEY to OLD_KEY (the source is still in the _OLD slot).
  3. Restart.
  4. Restore the DB from the Step 1 snapshot if Step 3 committed to the wrong tables: 
    sudo bash /opt/kesles/scripts/db_restore.sh \
      --backup pre-master-key-rotation
  5. Investigate the root cause before trying again.

Full Checklist​

  • Pre-flight: count enc:v1:% rows in 3 columns. Record the numbers.
  • Step 1: back up env value + snapshot DB. Verify backup file.
  • Step 2: generate the new key. Set _OLD + _NEW in env. No restart.
  • Step 3: run psp-rotate-key --dry-run → numbers match → --commit. Verify post-commit count matches.
  • Step 4: cut over KESLES_SECRET_ENCRYPTION_KEY to the new key. Restart. Smoke test PSP Endpoint Tester → 200.
  • Tail logs 24-48 hours, monitor decrypt errors.
  • Drop KESLES_SECRET_ENCRYPTION_KEY_OLD. Final restart.
  • Update internal password manager (Bitwarden / 1Password) — entry KESLES_SECRET_ENCRYPTION_KEY (production) swapped to the new key.
  • Update the incident log in ops/secret-rotation-log.md (date, reason, executor).

FAQ​

Q: Does the dev environment also need to be rotated? Not a priority. The dev DB usually doesn't hold business-valuable secrets. But it's recommended to use a key separate from prod (do not re-use values across envs) — even if the dev key leaks, prod is not automatically compromised.

Q: Can I skip Step 3 and just restart with the new key? No. At startup the service only validates the key format via loadMasterKey() (server.go, around line 136) — it does not fail on a wrong-but-valid key, because the row-decrypt check was retired when psp.api_keys was dropped from db_kesles_merchant (mig 088; psp.api_keys is now sole SOT in payment_service). With a wrong key the service starts fine but every PSP/acquirer decrypt fails at request time — gcm decrypt: message authentication failed. Re-encrypting in Step 3 is the only safe path.

Q: Do we need traffic downtime? Ideally not — Step 3 runs on a separate DB connection with a transaction per table. But the Step 4 restart window (~10 seconds) can drop in-flight requests; put up a maintenance banner if the audience is sensitive.

Q: What if there are rows in psp.api_keys not prefixed with enc:v1:? Those are legacy plaintext. psp-rotate-key must skip + log a warning. Do not re-encrypt because it is ambiguous (it might be intentional plaintext). Clean up manually via cmd/psp-encrypt before rotation.

Source References​