docs: add bulk incident import design spec

This commit is contained in:
2026-08-16 14:50:25 +08:00
parent dbff55dd7a
commit f68832bf54
@@ -0,0 +1,289 @@
# Bulk Incident Import — Design Specification
**Date:** 2026-08-16
**Status:** Ready for Implementation
**Scope:** Admin-only bulk import of historical incident cases with AI-extracted metadata
---
## Overview
**Problem:** Hundreds of historical incident case folders (containing PDFs, images, videos) need to be migrated into IMS as closed incidents for record-keeping and archival.
**Solution:** Bulk import system that accepts single-folder or .zip-based uploads, analyzes PDFs via DeepSeek API to extract incident metadata, and creates closed incidents with attached evidence files.
**Success Criteria:**
- Admin can upload single folder or .zip containing multiple case folders
- System auto-extracts metadata from PDFs (description, date, location, type, severity, witnesses)
- Admin reviews extracted data, fills required gaps, approves batch
- All incidents created as "Closed" status with full evidence file chain
- Audit log captures bulk import source for each incident
---
## Architecture
### Components
#### 1. Upload Handler (`POST /api/admin/bulk-import`)
- Accept multipart/form-data with single folder upload OR .zip file
- Extract files, group by folder (each folder = one incident)
- Validate: at least one file per folder, PDFs present for analysis
- Response: list of folders ready for metadata extraction
#### 2. PDF Metadata Analyzer
- Read PDF/DOC content (text extraction via file-type detection)
- Send text to DeepSeek API with structured extraction prompt
- Extract fields: description, date, location (site + zone), incident_type, priority, witnesses
- Return structured JSON for each folder's PDFs
- Cache results during session (don't re-analyze on retry)
**DeepSeek Prompt Template:**
```
Analyze the following incident document(s) and extract:
- Description: summary of what happened
- Date: incident date (YYYY-MM-DD format)
- Location: site name and zone/area
- Type: incident category (env/asset/security/fire)
- Priority: severity level (low/med/high)
- Witnesses: names and contact info if present
Document text:
[PDF content here]
Return JSON format:
{
"description": "...",
"date": "YYYY-MM-DD",
"location": {
"site": "...",
"zone": "..."
},
"incident_type": "env|asset|security|fire",
"priority": "low|med|high",
"witnesses": [{"name": "...", "contact": "..."}, ...]
}
```
#### 3. Preview & Approval Table
- Display extracted data as editable table
- Columns: folder name, description, date, location (site), location (zone), type, priority, # witnesses, # files
- Allow inline editing for any field (admin fills gaps)
- Mark rows as complete/incomplete (red if required field missing)
- "Import All" button only enabled when all rows have description + date + location
#### 4. Incident Factory
- Create incident from row data using `withUser()` transaction
- Set status = "Closed" (not "Open")
- Set created_by = current admin user
- Store bulk_import_source in audit_log old_value for tracking
- Attach all folder files as evidence (PDFs, images, videos)
- Generate unique incident reference_no (existing format: SITE-YYYYMM-####)
#### 5. Evidence Attachment
- For each file in folder: call `uploadEvidenceFile()` (existing helper)
- Store in `/var/lib/ims-evidence/[incident_id]/[filename]`
- Record in `evidence_files` table (same as regular incidents)
- No special staging/approval; all files attached as "verified" evidence
---
## Data Model
### New Table: `bulk_import_sessions` (optional, for audit)
```sql
CREATE TABLE bulk_import_sessions (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
imported_by UUID NOT NULL REFERENCES users(id),
upload_filename TEXT NOT NULL,
folder_count INT NOT NULL,
incident_count INT NOT NULL,
file_count INT NOT NULL,
created_at TIMESTAMP DEFAULT now()
);
```
### Existing Tables Modified
- `incidents`: use existing columns (description, created_by, status, etc.)
- `evidence_files`: attach files under incident_id (existing behavior)
- `audit_log`: new entries for each bulk-imported incident with `action='bulk_import'`
---
## User Flow
1. **Upload:** Admin navigates to `/admin/bulk-import`, uploads single folder or .zip
2. **Analysis:** System extracts files, analyzes PDFs via DeepSeek, shows progress
3. **Preview:** Table displays extracted metadata; admin reviews and fills gaps
4. **Approval:** Admin clicks "Import All" (button disabled if any required field missing)
5. **Creation:** System creates all incidents + attaches files in parallel
6. **Confirmation:** Shows summary (X incidents created, Y total files imported)
---
## API Routes
### `POST /api/admin/bulk-import` — Upload & Analyze
Request:
```
multipart/form-data:
file: <single-folder.zip or folder>
(or individual folder via file picker)
```
Response:
```json
{
"status": "success",
"folders": [
{
"id": "folder_hash",
"folder_name": "10AUG2026 - SUNIL",
"extracted": {
"description": "...",
"date": "2026-08-10",
"location": { "site": "Main WH", "zone": "Zone 7" },
"incident_type": "asset",
"priority": "high",
"witnesses": [...]
},
"files": {
"pdfs": 2,
"images": 3,
"videos": 1
},
"metadata_confidence": 0.92
}
]
}
```
### `POST /api/admin/bulk-import/create` — Batch Create
Request:
```json
{
"folders": [
{
"id": "folder_hash",
"description": "...",
"date": "2026-08-10",
"location": { "site": "...", "zone": "..." },
"incident_type": "asset",
"priority": "high",
"witnesses": [...]
}
]
}
```
Response:
```json
{
"status": "success",
"created_incidents": [
{
"id": "inc-123",
"reference_no": "MAIN-202608-0001",
"description": "...",
"status": "closed",
"file_count": 6
}
],
"total_files_imported": 42
}
```
---
## UI Page: `/admin/bulk-import`
**Layout:**
- Header: "Bulk Import Historical Incidents"
- Upload section: folder/zip picker + drag-drop zone
- Progress bar during analysis
- Preview table (editable)
- "Import All" button + confirmation modal
- Success summary with incident list
**Permissions:** Admin role only (`requireAdmin()`)
---
## Error Handling
| Scenario | Behavior |
|----------|----------|
| No PDFs in folder | Skip folder, warn admin ("No analyzable documents in 'folder_name'") |
| DeepSeek API fails | Show error, allow manual entry (fields stay empty) |
| Missing required field after manual review | Disable "Import" button, highlight row in red |
| File upload too large (>100MB total) | Reject, suggest splitting into multiple uploads |
| Folder name conflicts with existing incident | Auto-increment reference_no, notify admin |
| Database transaction fails on create | Rollback entire batch, show error with failed folder names |
---
## Testing Strategy
**Unit Tests:**
- PDF text extraction (mock file reads)
- DeepSeek prompt generation and JSON parsing
- Incident factory (create incident, attach evidence, audit log)
- Reference_no generation under collision scenarios
**Integration Tests:**
- End-to-end single-folder upload → preview → create
- Batch .zip upload with 5+ folders
- Missing required fields → error handling
- Evidence file attachment verification
**Manual Testing:**
- Upload real historical case folder
- Verify extracted metadata accuracy
- Confirm incident + evidence files appear in HSE incidents list
- Check audit_log for bulk_import entries
---
## Deployment Notes
**Environment:**
- `DEEPSEEK_API_KEY` must be set (already configured for risk analysis)
- `EVIDENCE_DIR` must exist and be writable (already set)
- Supabase storage for temporary zip extraction (use `/tmp` or disk)
**Performance:**
- DeepSeek analysis runs sequentially per folder (safe for hundreds)
- Incident creation (batch) can run in parallel to save time
- Estimated time: ~5-10 minutes for 100 folders (depends on PDF sizes + DeepSeek latency)
**Compliance:**
- Audit log tracks every bulk import (action='bulk_import', old_value=session_id)
- Closed status prevents accidental edits (no workflow triggers)
- Evidence retention policy applies (5-year minimum per DOSH)
---
## Future Enhancements (out of scope)
- Async job queue for very large imports (1000+ folders)
- OCR preprocessing for scanned PDFs
- Duplicate detection (warn if folder name matches existing incident)
- Scheduled imports (batch upload schedule)
---
## Files to Create/Modify
**Create:**
- `app/(protected)/admin/bulk-import/page.tsx` — admin page
- `components/admin/bulk-import-form.tsx` — upload + preview component
- `lib/incidents/bulk-import.ts` — metadata extraction + incident factory
- `app/api/admin/bulk-import/route.ts` — upload handler
- `app/api/admin/bulk-import/create/route.ts` — batch creation endpoint
**Modify:**
- `app/(protected)/admin/page.tsx` — add link to bulk import
- `lib/storage/evidence.ts` — no changes (use existing `uploadEvidenceFile()`)
**Optional:**
- `supabase/migrations/####_bulk_import_sessions.sql` — audit table (if needed)