docs: phase 9 — update vps-cron.md, add cutover runbook

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-24 06:35:15 +08:00
parent b3811b3633
commit 8acca41ca1
2 changed files with 138 additions and 9 deletions
@@ -0,0 +1,86 @@
+# Phase 9 — Cutover Runbook
+
+## Pre-flight checklist (before cutting traffic)
+
+Run from the project root on your local machine.
+
+- [ ] `bash db/01-load-schema.sh` — schema + RLS loaded on new server (14 tables)
+- [ ] `SUPABASE_DB_URL="..." bash db/02-dump-and-restore.sh` — all public data restored
+- [ ] `SUPABASE_DB_URL="..." bash db/03-migrate-passwords.sh` — passwords migrated, 0 NULL rows
+- [ ] `SUPABASE_URL="..." SUPABASE_SERVICE_ROLE_KEY="..." bash db/04-migrate-evidence.sh` — evidence files on server
+- [ ] `SUPABASE_DB_URL="..." bash db/05-verify.sh` — row counts match, embeddings present
+- [ ] `/var/www/ims/.env` populated on server (see `docs/new-server-setup.md` section I)
+- [ ] `bash deploy.sh` — Next.js standalone build deployed, `systemctl status ims` shows Active
+- [ ] nginx config live at `ims.setia.com.my` (see `docs/new-server-setup.md` sections E–F)
+- [ ] TLS cert issued (`sudo certbot --nginx -d ims.setia.com.my`)
+- [ ] `curl -I https://ims.setia.com.my/` returns HTTP/2 200
+- [ ] Cron jobs registered (see `docs/vps-cron.md`)
+- [ ] Daily backup cron registered (see `docs/new-server-setup.md` section K)
+
+## DNS cutover
+
+Point `ims.setia.com.my` A record to the new server's public IP.
+TTL: set to 60s before cutover, restore to 3600s after.
+
+## Smoke tests (run immediately after cutover)
+
+Test as each role. All must pass before telling users to switch.
+
+### Reporter role
+- [ ] Log in with a migrated user's real password — succeeds
+- [ ] Submit new incident with photo evidence — reference number generated, evidence stored
+- [ ] Open own incident detail — evidence renders (HMAC-signed URL serves file)
+
+### Supervisor role
+- [ ] Log in
+- [ ] View site incidents list — all historical incidents present
+- [ ] Triage an incident — triage saved, audit_log row written with correct `changed_by`
+
+### HSE role
+- [ ] Log in
+- [ ] Run AI triage suggestion — response received (Anthropic key working)
+- [ ] Run Similar Incidents — embeddings returning results
+- [ ] Open dashboard — stats render, JKKP8 export downloads CSV
+- [ ] AI risk flags panel loads
+
+### Admin role
+- [ ] Log in
+- [ ] Create new user — user created, no Supabase call
+- [ ] Delete user — user removed (auth + profile)
+
+### CAPA owner role
+- [ ] Log in
+- [ ] View assigned CAPAs
+- [ ] Mark a CAPA in-progress — status updates
+
+### Password reset flow
+- [ ] Request reset for own email — Brevo email arrives
+- [ ] Complete reset — new password accepted
+
+### Cron (manual trigger)
+- [ ] CAPA escalation endpoint returns `{"ok":true}`
+- [ ] Effectiveness recheck endpoint returns `{"ok":true}`
+
+## RLS spot-check (security)
+
+```bash
+# As app_user role with a reporter's user_id set:
+sudo -u postgres psql -d ims
+SET app.user_id = '<reporter-A-uuid>';
+SELECT * FROM incidents WHERE reported_by != '<reporter-A-uuid>';
+-- Expected: 0 rows (RLS blocks cross-user read)
+```
+
+## Post-cutover
+
+- [ ] Monitor `/var/log/ims-cron.log` for first cron run (08:00 MYT next day)
+- [ ] Monitor `sudo journalctl -u ims -f` for runtime errors
+- [ ] After 48h stable: take snapshot of new server DB (`pg_dump ims > /var/lib/ims/backups/pre-supabase-decommission.sql`)
+- [ ] Decommission Supabase project only after snapshot confirmed
+
+## Rollback plan
+
+If cutover fails within the first hour:
+1. Repoint DNS A record back to the old VPS IP (`64.176.82.100`)
+2. Supabase project remains live throughout — no data risk
+3. Investigate failure before re-attempting