docs: add bulk incident import design spec
This commit is contained in:
@@ -0,0 +1,289 @@
|
|||||||
|
# Bulk Incident Import — Design Specification
|
||||||
|
|
||||||
|
**Date:** 2026-08-16
|
||||||
|
**Status:** Ready for Implementation
|
||||||
|
**Scope:** Admin-only bulk import of historical incident cases with AI-extracted metadata
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Overview
|
||||||
|
|
||||||
|
**Problem:** Hundreds of historical incident case folders (containing PDFs, images, videos) need to be migrated into IMS as closed incidents for record-keeping and archival.
|
||||||
|
|
||||||
|
**Solution:** Bulk import system that accepts single-folder or .zip-based uploads, analyzes PDFs via DeepSeek API to extract incident metadata, and creates closed incidents with attached evidence files.
|
||||||
|
|
||||||
|
**Success Criteria:**
|
||||||
|
- Admin can upload single folder or .zip containing multiple case folders
|
||||||
|
- System auto-extracts metadata from PDFs (description, date, location, type, severity, witnesses)
|
||||||
|
- Admin reviews extracted data, fills required gaps, approves batch
|
||||||
|
- All incidents created as "Closed" status with full evidence file chain
|
||||||
|
- Audit log captures bulk import source for each incident
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Architecture
|
||||||
|
|
||||||
|
### Components
|
||||||
|
|
||||||
|
#### 1. Upload Handler (`POST /api/admin/bulk-import`)
|
||||||
|
- Accept multipart/form-data with single folder upload OR .zip file
|
||||||
|
- Extract files, group by folder (each folder = one incident)
|
||||||
|
- Validate: at least one file per folder, PDFs present for analysis
|
||||||
|
- Response: list of folders ready for metadata extraction
|
||||||
|
|
||||||
|
#### 2. PDF Metadata Analyzer
|
||||||
|
- Read PDF/DOC content (text extraction via file-type detection)
|
||||||
|
- Send text to DeepSeek API with structured extraction prompt
|
||||||
|
- Extract fields: description, date, location (site + zone), incident_type, priority, witnesses
|
||||||
|
- Return structured JSON for each folder's PDFs
|
||||||
|
- Cache results during session (don't re-analyze on retry)
|
||||||
|
|
||||||
|
**DeepSeek Prompt Template:**
|
||||||
|
```
|
||||||
|
Analyze the following incident document(s) and extract:
|
||||||
|
- Description: summary of what happened
|
||||||
|
- Date: incident date (YYYY-MM-DD format)
|
||||||
|
- Location: site name and zone/area
|
||||||
|
- Type: incident category (env/asset/security/fire)
|
||||||
|
- Priority: severity level (low/med/high)
|
||||||
|
- Witnesses: names and contact info if present
|
||||||
|
|
||||||
|
Document text:
|
||||||
|
[PDF content here]
|
||||||
|
|
||||||
|
Return JSON format:
|
||||||
|
{
|
||||||
|
"description": "...",
|
||||||
|
"date": "YYYY-MM-DD",
|
||||||
|
"location": {
|
||||||
|
"site": "...",
|
||||||
|
"zone": "..."
|
||||||
|
},
|
||||||
|
"incident_type": "env|asset|security|fire",
|
||||||
|
"priority": "low|med|high",
|
||||||
|
"witnesses": [{"name": "...", "contact": "..."}, ...]
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
#### 3. Preview & Approval Table
|
||||||
|
- Display extracted data as editable table
|
||||||
|
- Columns: folder name, description, date, location (site), location (zone), type, priority, # witnesses, # files
|
||||||
|
- Allow inline editing for any field (admin fills gaps)
|
||||||
|
- Mark rows as complete/incomplete (red if required field missing)
|
||||||
|
- "Import All" button only enabled when all rows have description + date + location
|
||||||
|
|
||||||
|
#### 4. Incident Factory
|
||||||
|
- Create incident from row data using `withUser()` transaction
|
||||||
|
- Set status = "Closed" (not "Open")
|
||||||
|
- Set created_by = current admin user
|
||||||
|
- Store bulk_import_source in audit_log old_value for tracking
|
||||||
|
- Attach all folder files as evidence (PDFs, images, videos)
|
||||||
|
- Generate unique incident reference_no (existing format: SITE-YYYYMM-####)
|
||||||
|
|
||||||
|
#### 5. Evidence Attachment
|
||||||
|
- For each file in folder: call `uploadEvidenceFile()` (existing helper)
|
||||||
|
- Store in `/var/lib/ims-evidence/[incident_id]/[filename]`
|
||||||
|
- Record in `evidence_files` table (same as regular incidents)
|
||||||
|
- No special staging/approval; all files attached as "verified" evidence
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Data Model
|
||||||
|
|
||||||
|
### New Table: `bulk_import_sessions` (optional, for audit)
|
||||||
|
```sql
|
||||||
|
CREATE TABLE bulk_import_sessions (
|
||||||
|
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
|
||||||
|
imported_by UUID NOT NULL REFERENCES users(id),
|
||||||
|
upload_filename TEXT NOT NULL,
|
||||||
|
folder_count INT NOT NULL,
|
||||||
|
incident_count INT NOT NULL,
|
||||||
|
file_count INT NOT NULL,
|
||||||
|
created_at TIMESTAMP DEFAULT now()
|
||||||
|
);
|
||||||
|
```
|
||||||
|
|
||||||
|
### Existing Tables Modified
|
||||||
|
- `incidents`: use existing columns (description, created_by, status, etc.)
|
||||||
|
- `evidence_files`: attach files under incident_id (existing behavior)
|
||||||
|
- `audit_log`: new entries for each bulk-imported incident with `action='bulk_import'`
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## User Flow
|
||||||
|
|
||||||
|
1. **Upload:** Admin navigates to `/admin/bulk-import`, uploads single folder or .zip
|
||||||
|
2. **Analysis:** System extracts files, analyzes PDFs via DeepSeek, shows progress
|
||||||
|
3. **Preview:** Table displays extracted metadata; admin reviews and fills gaps
|
||||||
|
4. **Approval:** Admin clicks "Import All" (button disabled if any required field missing)
|
||||||
|
5. **Creation:** System creates all incidents + attaches files in parallel
|
||||||
|
6. **Confirmation:** Shows summary (X incidents created, Y total files imported)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## API Routes
|
||||||
|
|
||||||
|
### `POST /api/admin/bulk-import` — Upload & Analyze
|
||||||
|
Request:
|
||||||
|
```
|
||||||
|
multipart/form-data:
|
||||||
|
file: <single-folder.zip or folder>
|
||||||
|
(or individual folder via file picker)
|
||||||
|
```
|
||||||
|
|
||||||
|
Response:
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"status": "success",
|
||||||
|
"folders": [
|
||||||
|
{
|
||||||
|
"id": "folder_hash",
|
||||||
|
"folder_name": "10AUG2026 - SUNIL",
|
||||||
|
"extracted": {
|
||||||
|
"description": "...",
|
||||||
|
"date": "2026-08-10",
|
||||||
|
"location": { "site": "Main WH", "zone": "Zone 7" },
|
||||||
|
"incident_type": "asset",
|
||||||
|
"priority": "high",
|
||||||
|
"witnesses": [...]
|
||||||
|
},
|
||||||
|
"files": {
|
||||||
|
"pdfs": 2,
|
||||||
|
"images": 3,
|
||||||
|
"videos": 1
|
||||||
|
},
|
||||||
|
"metadata_confidence": 0.92
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### `POST /api/admin/bulk-import/create` — Batch Create
|
||||||
|
Request:
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"folders": [
|
||||||
|
{
|
||||||
|
"id": "folder_hash",
|
||||||
|
"description": "...",
|
||||||
|
"date": "2026-08-10",
|
||||||
|
"location": { "site": "...", "zone": "..." },
|
||||||
|
"incident_type": "asset",
|
||||||
|
"priority": "high",
|
||||||
|
"witnesses": [...]
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
Response:
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"status": "success",
|
||||||
|
"created_incidents": [
|
||||||
|
{
|
||||||
|
"id": "inc-123",
|
||||||
|
"reference_no": "MAIN-202608-0001",
|
||||||
|
"description": "...",
|
||||||
|
"status": "closed",
|
||||||
|
"file_count": 6
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"total_files_imported": 42
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## UI Page: `/admin/bulk-import`
|
||||||
|
|
||||||
|
**Layout:**
|
||||||
|
- Header: "Bulk Import Historical Incidents"
|
||||||
|
- Upload section: folder/zip picker + drag-drop zone
|
||||||
|
- Progress bar during analysis
|
||||||
|
- Preview table (editable)
|
||||||
|
- "Import All" button + confirmation modal
|
||||||
|
- Success summary with incident list
|
||||||
|
|
||||||
|
**Permissions:** Admin role only (`requireAdmin()`)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Error Handling
|
||||||
|
|
||||||
|
| Scenario | Behavior |
|
||||||
|
|----------|----------|
|
||||||
|
| No PDFs in folder | Skip folder, warn admin ("No analyzable documents in 'folder_name'") |
|
||||||
|
| DeepSeek API fails | Show error, allow manual entry (fields stay empty) |
|
||||||
|
| Missing required field after manual review | Disable "Import" button, highlight row in red |
|
||||||
|
| File upload too large (>100MB total) | Reject, suggest splitting into multiple uploads |
|
||||||
|
| Folder name conflicts with existing incident | Auto-increment reference_no, notify admin |
|
||||||
|
| Database transaction fails on create | Rollback entire batch, show error with failed folder names |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Testing Strategy
|
||||||
|
|
||||||
|
**Unit Tests:**
|
||||||
|
- PDF text extraction (mock file reads)
|
||||||
|
- DeepSeek prompt generation and JSON parsing
|
||||||
|
- Incident factory (create incident, attach evidence, audit log)
|
||||||
|
- Reference_no generation under collision scenarios
|
||||||
|
|
||||||
|
**Integration Tests:**
|
||||||
|
- End-to-end single-folder upload → preview → create
|
||||||
|
- Batch .zip upload with 5+ folders
|
||||||
|
- Missing required fields → error handling
|
||||||
|
- Evidence file attachment verification
|
||||||
|
|
||||||
|
**Manual Testing:**
|
||||||
|
- Upload real historical case folder
|
||||||
|
- Verify extracted metadata accuracy
|
||||||
|
- Confirm incident + evidence files appear in HSE incidents list
|
||||||
|
- Check audit_log for bulk_import entries
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Deployment Notes
|
||||||
|
|
||||||
|
**Environment:**
|
||||||
|
- `DEEPSEEK_API_KEY` must be set (already configured for risk analysis)
|
||||||
|
- `EVIDENCE_DIR` must exist and be writable (already set)
|
||||||
|
- Supabase storage for temporary zip extraction (use `/tmp` or disk)
|
||||||
|
|
||||||
|
**Performance:**
|
||||||
|
- DeepSeek analysis runs sequentially per folder (safe for hundreds)
|
||||||
|
- Incident creation (batch) can run in parallel to save time
|
||||||
|
- Estimated time: ~5-10 minutes for 100 folders (depends on PDF sizes + DeepSeek latency)
|
||||||
|
|
||||||
|
**Compliance:**
|
||||||
|
- Audit log tracks every bulk import (action='bulk_import', old_value=session_id)
|
||||||
|
- Closed status prevents accidental edits (no workflow triggers)
|
||||||
|
- Evidence retention policy applies (5-year minimum per DOSH)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Future Enhancements (out of scope)
|
||||||
|
|
||||||
|
- Async job queue for very large imports (1000+ folders)
|
||||||
|
- OCR preprocessing for scanned PDFs
|
||||||
|
- Duplicate detection (warn if folder name matches existing incident)
|
||||||
|
- Scheduled imports (batch upload schedule)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Files to Create/Modify
|
||||||
|
|
||||||
|
**Create:**
|
||||||
|
- `app/(protected)/admin/bulk-import/page.tsx` — admin page
|
||||||
|
- `components/admin/bulk-import-form.tsx` — upload + preview component
|
||||||
|
- `lib/incidents/bulk-import.ts` — metadata extraction + incident factory
|
||||||
|
- `app/api/admin/bulk-import/route.ts` — upload handler
|
||||||
|
- `app/api/admin/bulk-import/create/route.ts` — batch creation endpoint
|
||||||
|
|
||||||
|
**Modify:**
|
||||||
|
- `app/(protected)/admin/page.tsx` — add link to bulk import
|
||||||
|
- `lib/storage/evidence.ts` — no changes (use existing `uploadEvidenceFile()`)
|
||||||
|
|
||||||
|
**Optional:**
|
||||||
|
- `supabase/migrations/####_bulk_import_sessions.sql` — audit table (if needed)
|
||||||
Reference in New Issue
Block a user