diff --git a/docs/superpowers/specs/2026-08-16-bulk-incident-import-design.md b/docs/superpowers/specs/2026-08-16-bulk-incident-import-design.md new file mode 100644 index 0000000..327223e --- /dev/null +++ b/docs/superpowers/specs/2026-08-16-bulk-incident-import-design.md @@ -0,0 +1,289 @@ +# Bulk Incident Import — Design Specification + +**Date:** 2026-08-16 +**Status:** Ready for Implementation +**Scope:** Admin-only bulk import of historical incident cases with AI-extracted metadata + +--- + +## Overview + +**Problem:** Hundreds of historical incident case folders (containing PDFs, images, videos) need to be migrated into IMS as closed incidents for record-keeping and archival. + +**Solution:** Bulk import system that accepts single-folder or .zip-based uploads, analyzes PDFs via DeepSeek API to extract incident metadata, and creates closed incidents with attached evidence files. + +**Success Criteria:** +- Admin can upload single folder or .zip containing multiple case folders +- System auto-extracts metadata from PDFs (description, date, location, type, severity, witnesses) +- Admin reviews extracted data, fills required gaps, approves batch +- All incidents created as "Closed" status with full evidence file chain +- Audit log captures bulk import source for each incident + +--- + +## Architecture + +### Components + +#### 1. Upload Handler (`POST /api/admin/bulk-import`) +- Accept multipart/form-data with single folder upload OR .zip file +- Extract files, group by folder (each folder = one incident) +- Validate: at least one file per folder, PDFs present for analysis +- Response: list of folders ready for metadata extraction + +#### 2. PDF Metadata Analyzer +- Read PDF/DOC content (text extraction via file-type detection) +- Send text to DeepSeek API with structured extraction prompt +- Extract fields: description, date, location (site + zone), incident_type, priority, witnesses +- Return structured JSON for each folder's PDFs +- Cache results during session (don't re-analyze on retry) + +**DeepSeek Prompt Template:** +``` +Analyze the following incident document(s) and extract: +- Description: summary of what happened +- Date: incident date (YYYY-MM-DD format) +- Location: site name and zone/area +- Type: incident category (env/asset/security/fire) +- Priority: severity level (low/med/high) +- Witnesses: names and contact info if present + +Document text: +[PDF content here] + +Return JSON format: +{ + "description": "...", + "date": "YYYY-MM-DD", + "location": { + "site": "...", + "zone": "..." + }, + "incident_type": "env|asset|security|fire", + "priority": "low|med|high", + "witnesses": [{"name": "...", "contact": "..."}, ...] +} +``` + +#### 3. Preview & Approval Table +- Display extracted data as editable table +- Columns: folder name, description, date, location (site), location (zone), type, priority, # witnesses, # files +- Allow inline editing for any field (admin fills gaps) +- Mark rows as complete/incomplete (red if required field missing) +- "Import All" button only enabled when all rows have description + date + location + +#### 4. Incident Factory +- Create incident from row data using `withUser()` transaction +- Set status = "Closed" (not "Open") +- Set created_by = current admin user +- Store bulk_import_source in audit_log old_value for tracking +- Attach all folder files as evidence (PDFs, images, videos) +- Generate unique incident reference_no (existing format: SITE-YYYYMM-####) + +#### 5. Evidence Attachment +- For each file in folder: call `uploadEvidenceFile()` (existing helper) +- Store in `/var/lib/ims-evidence/[incident_id]/[filename]` +- Record in `evidence_files` table (same as regular incidents) +- No special staging/approval; all files attached as "verified" evidence + +--- + +## Data Model + +### New Table: `bulk_import_sessions` (optional, for audit) +```sql +CREATE TABLE bulk_import_sessions ( + id UUID PRIMARY KEY DEFAULT gen_random_uuid(), + imported_by UUID NOT NULL REFERENCES users(id), + upload_filename TEXT NOT NULL, + folder_count INT NOT NULL, + incident_count INT NOT NULL, + file_count INT NOT NULL, + created_at TIMESTAMP DEFAULT now() +); +``` + +### Existing Tables Modified +- `incidents`: use existing columns (description, created_by, status, etc.) +- `evidence_files`: attach files under incident_id (existing behavior) +- `audit_log`: new entries for each bulk-imported incident with `action='bulk_import'` + +--- + +## User Flow + +1. **Upload:** Admin navigates to `/admin/bulk-import`, uploads single folder or .zip +2. **Analysis:** System extracts files, analyzes PDFs via DeepSeek, shows progress +3. **Preview:** Table displays extracted metadata; admin reviews and fills gaps +4. **Approval:** Admin clicks "Import All" (button disabled if any required field missing) +5. **Creation:** System creates all incidents + attaches files in parallel +6. **Confirmation:** Shows summary (X incidents created, Y total files imported) + +--- + +## API Routes + +### `POST /api/admin/bulk-import` — Upload & Analyze +Request: +``` +multipart/form-data: + file: + (or individual folder via file picker) +``` + +Response: +```json +{ + "status": "success", + "folders": [ + { + "id": "folder_hash", + "folder_name": "10AUG2026 - SUNIL", + "extracted": { + "description": "...", + "date": "2026-08-10", + "location": { "site": "Main WH", "zone": "Zone 7" }, + "incident_type": "asset", + "priority": "high", + "witnesses": [...] + }, + "files": { + "pdfs": 2, + "images": 3, + "videos": 1 + }, + "metadata_confidence": 0.92 + } + ] +} +``` + +### `POST /api/admin/bulk-import/create` — Batch Create +Request: +```json +{ + "folders": [ + { + "id": "folder_hash", + "description": "...", + "date": "2026-08-10", + "location": { "site": "...", "zone": "..." }, + "incident_type": "asset", + "priority": "high", + "witnesses": [...] + } + ] +} +``` + +Response: +```json +{ + "status": "success", + "created_incidents": [ + { + "id": "inc-123", + "reference_no": "MAIN-202608-0001", + "description": "...", + "status": "closed", + "file_count": 6 + } + ], + "total_files_imported": 42 +} +``` + +--- + +## UI Page: `/admin/bulk-import` + +**Layout:** +- Header: "Bulk Import Historical Incidents" +- Upload section: folder/zip picker + drag-drop zone +- Progress bar during analysis +- Preview table (editable) +- "Import All" button + confirmation modal +- Success summary with incident list + +**Permissions:** Admin role only (`requireAdmin()`) + +--- + +## Error Handling + +| Scenario | Behavior | +|----------|----------| +| No PDFs in folder | Skip folder, warn admin ("No analyzable documents in 'folder_name'") | +| DeepSeek API fails | Show error, allow manual entry (fields stay empty) | +| Missing required field after manual review | Disable "Import" button, highlight row in red | +| File upload too large (>100MB total) | Reject, suggest splitting into multiple uploads | +| Folder name conflicts with existing incident | Auto-increment reference_no, notify admin | +| Database transaction fails on create | Rollback entire batch, show error with failed folder names | + +--- + +## Testing Strategy + +**Unit Tests:** +- PDF text extraction (mock file reads) +- DeepSeek prompt generation and JSON parsing +- Incident factory (create incident, attach evidence, audit log) +- Reference_no generation under collision scenarios + +**Integration Tests:** +- End-to-end single-folder upload → preview → create +- Batch .zip upload with 5+ folders +- Missing required fields → error handling +- Evidence file attachment verification + +**Manual Testing:** +- Upload real historical case folder +- Verify extracted metadata accuracy +- Confirm incident + evidence files appear in HSE incidents list +- Check audit_log for bulk_import entries + +--- + +## Deployment Notes + +**Environment:** +- `DEEPSEEK_API_KEY` must be set (already configured for risk analysis) +- `EVIDENCE_DIR` must exist and be writable (already set) +- Supabase storage for temporary zip extraction (use `/tmp` or disk) + +**Performance:** +- DeepSeek analysis runs sequentially per folder (safe for hundreds) +- Incident creation (batch) can run in parallel to save time +- Estimated time: ~5-10 minutes for 100 folders (depends on PDF sizes + DeepSeek latency) + +**Compliance:** +- Audit log tracks every bulk import (action='bulk_import', old_value=session_id) +- Closed status prevents accidental edits (no workflow triggers) +- Evidence retention policy applies (5-year minimum per DOSH) + +--- + +## Future Enhancements (out of scope) + +- Async job queue for very large imports (1000+ folders) +- OCR preprocessing for scanned PDFs +- Duplicate detection (warn if folder name matches existing incident) +- Scheduled imports (batch upload schedule) + +--- + +## Files to Create/Modify + +**Create:** +- `app/(protected)/admin/bulk-import/page.tsx` — admin page +- `components/admin/bulk-import-form.tsx` — upload + preview component +- `lib/incidents/bulk-import.ts` — metadata extraction + incident factory +- `app/api/admin/bulk-import/route.ts` — upload handler +- `app/api/admin/bulk-import/create/route.ts` — batch creation endpoint + +**Modify:** +- `app/(protected)/admin/page.tsx` — add link to bulk import +- `lib/storage/evidence.ts` — no changes (use existing `uploadEvidenceFile()`) + +**Optional:** +- `supabase/migrations/####_bulk_import_sessions.sql` — audit table (if needed)