docs: phase 9 — update vps-cron.md, add cutover runbook
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
Claude Sonnet 4.6
@@ -0,0 +1,86 @@
|
||||
# Phase 9 — Cutover Runbook
|
||||
|
||||
## Pre-flight checklist (before cutting traffic)
|
||||
|
||||
Run from the project root on your local machine.
|
||||
|
||||
- [ ] `bash db/01-load-schema.sh` — schema + RLS loaded on new server (14 tables)
|
||||
- [ ] `SUPABASE_DB_URL="..." bash db/02-dump-and-restore.sh` — all public data restored
|
||||
- [ ] `SUPABASE_DB_URL="..." bash db/03-migrate-passwords.sh` — passwords migrated, 0 NULL rows
|
||||
- [ ] `SUPABASE_URL="..." SUPABASE_SERVICE_ROLE_KEY="..." bash db/04-migrate-evidence.sh` — evidence files on server
|
||||
- [ ] `SUPABASE_DB_URL="..." bash db/05-verify.sh` — row counts match, embeddings present
|
||||
- [ ] `/var/www/ims/.env` populated on server (see `docs/new-server-setup.md` section I)
|
||||
- [ ] `bash deploy.sh` — Next.js standalone build deployed, `systemctl status ims` shows Active
|
||||
- [ ] nginx config live at `ims.setia.com.my` (see `docs/new-server-setup.md` sections E–F)
|
||||
- [ ] TLS cert issued (`sudo certbot --nginx -d ims.setia.com.my`)
|
||||
- [ ] `curl -I https://ims.setia.com.my/` returns HTTP/2 200
|
||||
- [ ] Cron jobs registered (see `docs/vps-cron.md`)
|
||||
- [ ] Daily backup cron registered (see `docs/new-server-setup.md` section K)
|
||||
|
||||
## DNS cutover
|
||||
|
||||
Point `ims.setia.com.my` A record to the new server's public IP.
|
||||
TTL: set to 60s before cutover, restore to 3600s after.
|
||||
|
||||
## Smoke tests (run immediately after cutover)
|
||||
|
||||
Test as each role. All must pass before telling users to switch.
|
||||
|
||||
### Reporter role
|
||||
- [ ] Log in with a migrated user's real password — succeeds
|
||||
- [ ] Submit new incident with photo evidence — reference number generated, evidence stored
|
||||
- [ ] Open own incident detail — evidence renders (HMAC-signed URL serves file)
|
||||
|
||||
### Supervisor role
|
||||
- [ ] Log in
|
||||
- [ ] View site incidents list — all historical incidents present
|
||||
- [ ] Triage an incident — triage saved, audit_log row written with correct `changed_by`
|
||||
|
||||
### HSE role
|
||||
- [ ] Log in
|
||||
- [ ] Run AI triage suggestion — response received (Anthropic key working)
|
||||
- [ ] Run Similar Incidents — embeddings returning results
|
||||
- [ ] Open dashboard — stats render, JKKP8 export downloads CSV
|
||||
- [ ] AI risk flags panel loads
|
||||
|
||||
### Admin role
|
||||
- [ ] Log in
|
||||
- [ ] Create new user — user created, no Supabase call
|
||||
- [ ] Delete user — user removed (auth + profile)
|
||||
|
||||
### CAPA owner role
|
||||
- [ ] Log in
|
||||
- [ ] View assigned CAPAs
|
||||
- [ ] Mark a CAPA in-progress — status updates
|
||||
|
||||
### Password reset flow
|
||||
- [ ] Request reset for own email — Brevo email arrives
|
||||
- [ ] Complete reset — new password accepted
|
||||
|
||||
### Cron (manual trigger)
|
||||
- [ ] CAPA escalation endpoint returns `{"ok":true}`
|
||||
- [ ] Effectiveness recheck endpoint returns `{"ok":true}`
|
||||
|
||||
## RLS spot-check (security)
|
||||
|
||||
```bash
|
||||
# As app_user role with a reporter's user_id set:
|
||||
sudo -u postgres psql -d ims
|
||||
SET app.user_id = '<reporter-A-uuid>';
|
||||
SELECT * FROM incidents WHERE reported_by != '<reporter-A-uuid>';
|
||||
-- Expected: 0 rows (RLS blocks cross-user read)
|
||||
```
|
||||
|
||||
## Post-cutover
|
||||
|
||||
- [ ] Monitor `/var/log/ims-cron.log` for first cron run (08:00 MYT next day)
|
||||
- [ ] Monitor `sudo journalctl -u ims -f` for runtime errors
|
||||
- [ ] After 48h stable: take snapshot of new server DB (`pg_dump ims > /var/lib/ims/backups/pre-supabase-decommission.sql`)
|
||||
- [ ] Decommission Supabase project only after snapshot confirmed
|
||||
|
||||
## Rollback plan
|
||||
|
||||
If cutover fails within the first hour:
|
||||
1. Repoint DNS A record back to the old VPS IP (`64.176.82.100`)
|
||||
2. Supabase project remains live throughout — no data risk
|
||||
3. Investigate failure before re-attempting
|
||||
+52
-9
@@ -1,22 +1,65 @@
|
||||
# VPS Cron Jobs
|
||||
# VPS Cron Jobs — ims.setia.com.my
|
||||
|
||||
## CAPA escalation
|
||||
App binds to `127.0.0.1:3003`. All cron calls hit that directly (no nginx proxy overhead).
|
||||
`CRON_SECRET` is in `/var/www/ims/.env`.
|
||||
|
||||
Runs daily at 00:00 UTC (08:00 MYT). Checks all open CAPAs against 4 thresholds
|
||||
(`warning_3d`, `due_today`, `overdue_3d`, `overdue_7d`) and sends email via Resend.
|
||||
## CAPA Overdue Escalation
|
||||
|
||||
Runs daily at 08:00 MYT (00:00 UTC). Checks all open CAPAs against 4 thresholds
|
||||
(`warning_3d`, `due_today`, `overdue_3d`, `overdue_7d`) and sends email via Brevo.
|
||||
Each threshold fires once per CAPA, tracked in `notifications_log.status`.
|
||||
|
||||
**Install (SSH into VPS):**
|
||||
**Install (SSH into server, then):**
|
||||
```bash
|
||||
crontab -e
|
||||
# Add:
|
||||
0 0 * * * curl -s -H "Authorization: Bearer $(grep CRON_SECRET /ims/.env.local | cut -d= -f2)" http://localhost:3000/ims/api/cron/capa-escalation >> /var/log/ims-cron.log 2>&1
|
||||
```
|
||||
Add:
|
||||
```
|
||||
0 0 * * * CRON_SECRET=$(grep ^CRON_SECRET /var/www/ims/.env | cut -d= -f2) curl -s -H "Authorization: Bearer $CRON_SECRET" http://127.0.0.1:3003/api/cron/capa-escalation >> /var/log/ims-cron.log 2>&1
|
||||
```
|
||||
|
||||
**Manual test:**
|
||||
```bash
|
||||
curl -s -H "Authorization: Bearer $CRON_SECRET" http://localhost:3000/ims/api/cron/capa-escalation
|
||||
CRON_SECRET=$(grep ^CRON_SECRET /var/www/ims/.env | cut -d= -f2)
|
||||
curl -s -H "Authorization: Bearer $CRON_SECRET" http://127.0.0.1:3003/api/cron/capa-escalation
|
||||
# Expected: {"ok":true,"notified":N}
|
||||
```
|
||||
|
||||
**Log:** `/var/log/ims-cron.log`
|
||||
---
|
||||
|
||||
## CAPA Effectiveness Recheck
|
||||
|
||||
Runs daily at 08:30 MYT (00:30 UTC). Re-evaluates CAPAs marked `pending_verification`
|
||||
past their recheck date and sends reminder email to the assigned verifier via Brevo.
|
||||
|
||||
**Install:**
|
||||
Add to crontab (same `crontab -e` session):
|
||||
```
|
||||
30 0 * * * CRON_SECRET=$(grep ^CRON_SECRET /var/www/ims/.env | cut -d= -f2) curl -s -H "Authorization: Bearer $CRON_SECRET" http://127.0.0.1:3003/api/cron/effectiveness-recheck >> /var/log/ims-cron.log 2>&1
|
||||
```
|
||||
|
||||
**Manual test:**
|
||||
```bash
|
||||
CRON_SECRET=$(grep ^CRON_SECRET /var/www/ims/.env | cut -d= -f2)
|
||||
curl -s -H "Authorization: Bearer $CRON_SECRET" http://127.0.0.1:3003/api/cron/effectiveness-recheck
|
||||
# Expected: {"ok":true}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Log Rotation
|
||||
|
||||
Prevent `/var/log/ims-cron.log` from growing unbounded:
|
||||
```bash
|
||||
sudo nano /etc/logrotate.d/ims-cron
|
||||
```
|
||||
Paste:
|
||||
```
|
||||
/var/log/ims-cron.log {
|
||||
daily
|
||||
rotate 30
|
||||
compress
|
||||
missingok
|
||||
notifempty
|
||||
}
|
||||
```
|
||||
|
||||
Reference in New Issue
Block a user