8.9 KiB
8.9 KiB
Bulk Incident Import — Design Specification
Date: 2026-08-16
Status: Ready for Implementation
Scope: Admin-only bulk import of historical incident cases with AI-extracted metadata
Overview
Problem: Hundreds of historical incident case folders (containing PDFs, images, videos) need to be migrated into IMS as closed incidents for record-keeping and archival.
Solution: Bulk import system that accepts single-folder or .zip-based uploads, analyzes PDFs via DeepSeek API to extract incident metadata, and creates closed incidents with attached evidence files.
Success Criteria:
- Admin can upload single folder or .zip containing multiple case folders
- System auto-extracts metadata from PDFs (description, date, location, type, severity, witnesses)
- Admin reviews extracted data, fills required gaps, approves batch
- All incidents created as "Closed" status with full evidence file chain
- Audit log captures bulk import source for each incident
Architecture
Components
1. Upload Handler (POST /api/admin/bulk-import)
- Accept multipart/form-data with single folder upload OR .zip file
- Extract files, group by folder (each folder = one incident)
- Validate: at least one file per folder, PDFs present for analysis
- Response: list of folders ready for metadata extraction
2. PDF Metadata Analyzer
- Read PDF/DOC content (text extraction via file-type detection)
- Send text to DeepSeek API with structured extraction prompt
- Extract fields: description, date, location (site + zone), incident_type, priority, witnesses
- Return structured JSON for each folder's PDFs
- Cache results during session (don't re-analyze on retry)
DeepSeek Prompt Template:
Analyze the following incident document(s) and extract:
- Description: summary of what happened
- Date: incident date (YYYY-MM-DD format)
- Location: site name and zone/area
- Type: incident category (env/asset/security/fire)
- Priority: severity level (low/med/high)
- Witnesses: names and contact info if present
Document text:
[PDF content here]
Return JSON format:
{
"description": "...",
"date": "YYYY-MM-DD",
"location": {
"site": "...",
"zone": "..."
},
"incident_type": "env|asset|security|fire",
"priority": "low|med|high",
"witnesses": [{"name": "...", "contact": "..."}, ...]
}
3. Preview & Approval Table
- Display extracted data as editable table
- Columns: folder name, description, date, location (site), location (zone), type, priority, # witnesses, # files
- Allow inline editing for any field (admin fills gaps)
- Mark rows as complete/incomplete (red if required field missing)
- "Import All" button only enabled when all rows have description + date + location
4. Incident Factory
- Create incident from row data using
withUser()transaction - Set status = "Closed" (not "Open")
- Set created_by = current admin user
- Store bulk_import_source in audit_log old_value for tracking
- Attach all folder files as evidence (PDFs, images, videos)
- Generate unique incident reference_no (existing format: SITE-YYYYMM-####)
5. Evidence Attachment
- For each file in folder: call
uploadEvidenceFile()(existing helper) - Store in
/var/lib/ims-evidence/[incident_id]/[filename] - Record in
evidence_filestable (same as regular incidents) - No special staging/approval; all files attached as "verified" evidence
Data Model
New Table: bulk_import_sessions (optional, for audit)
CREATE TABLE bulk_import_sessions (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
imported_by UUID NOT NULL REFERENCES users(id),
upload_filename TEXT NOT NULL,
folder_count INT NOT NULL,
incident_count INT NOT NULL,
file_count INT NOT NULL,
created_at TIMESTAMP DEFAULT now()
);
Existing Tables Modified
incidents: use existing columns (description, created_by, status, etc.)evidence_files: attach files under incident_id (existing behavior)audit_log: new entries for each bulk-imported incident withaction='bulk_import'
User Flow
- Upload: Admin navigates to
/admin/bulk-import, uploads single folder or .zip - Analysis: System extracts files, analyzes PDFs via DeepSeek, shows progress
- Preview: Table displays extracted metadata; admin reviews and fills gaps
- Approval: Admin clicks "Import All" (button disabled if any required field missing)
- Creation: System creates all incidents + attaches files in parallel
- Confirmation: Shows summary (X incidents created, Y total files imported)
API Routes
POST /api/admin/bulk-import — Upload & Analyze
Request:
multipart/form-data:
file: <single-folder.zip or folder>
(or individual folder via file picker)
Response:
{
"status": "success",
"folders": [
{
"id": "folder_hash",
"folder_name": "10AUG2026 - SUNIL",
"extracted": {
"description": "...",
"date": "2026-08-10",
"location": { "site": "Main WH", "zone": "Zone 7" },
"incident_type": "asset",
"priority": "high",
"witnesses": [...]
},
"files": {
"pdfs": 2,
"images": 3,
"videos": 1
},
"metadata_confidence": 0.92
}
]
}
POST /api/admin/bulk-import/create — Batch Create
Request:
{
"folders": [
{
"id": "folder_hash",
"description": "...",
"date": "2026-08-10",
"location": { "site": "...", "zone": "..." },
"incident_type": "asset",
"priority": "high",
"witnesses": [...]
}
]
}
Response:
{
"status": "success",
"created_incidents": [
{
"id": "inc-123",
"reference_no": "MAIN-202608-0001",
"description": "...",
"status": "closed",
"file_count": 6
}
],
"total_files_imported": 42
}
UI Page: /admin/bulk-import
Layout:
- Header: "Bulk Import Historical Incidents"
- Upload section: folder/zip picker + drag-drop zone
- Progress bar during analysis
- Preview table (editable)
- "Import All" button + confirmation modal
- Success summary with incident list
Permissions: Admin role only (requireAdmin())
Error Handling
| Scenario | Behavior |
|---|---|
| No PDFs in folder | Skip folder, warn admin ("No analyzable documents in 'folder_name'") |
| DeepSeek API fails | Show error, allow manual entry (fields stay empty) |
| Missing required field after manual review | Disable "Import" button, highlight row in red |
| File upload too large (>100MB total) | Reject, suggest splitting into multiple uploads |
| Folder name conflicts with existing incident | Auto-increment reference_no, notify admin |
| Database transaction fails on create | Rollback entire batch, show error with failed folder names |
Testing Strategy
Unit Tests:
- PDF text extraction (mock file reads)
- DeepSeek prompt generation and JSON parsing
- Incident factory (create incident, attach evidence, audit log)
- Reference_no generation under collision scenarios
Integration Tests:
- End-to-end single-folder upload → preview → create
- Batch .zip upload with 5+ folders
- Missing required fields → error handling
- Evidence file attachment verification
Manual Testing:
- Upload real historical case folder
- Verify extracted metadata accuracy
- Confirm incident + evidence files appear in HSE incidents list
- Check audit_log for bulk_import entries
Deployment Notes
Environment:
DEEPSEEK_API_KEYmust be set (already configured for risk analysis)EVIDENCE_DIRmust exist and be writable (already set)- Supabase storage for temporary zip extraction (use
/tmpor disk)
Performance:
- DeepSeek analysis runs sequentially per folder (safe for hundreds)
- Incident creation (batch) can run in parallel to save time
- Estimated time: ~5-10 minutes for 100 folders (depends on PDF sizes + DeepSeek latency)
Compliance:
- Audit log tracks every bulk import (action='bulk_import', old_value=session_id)
- Closed status prevents accidental edits (no workflow triggers)
- Evidence retention policy applies (5-year minimum per DOSH)
Future Enhancements (out of scope)
- Async job queue for very large imports (1000+ folders)
- OCR preprocessing for scanned PDFs
- Duplicate detection (warn if folder name matches existing incident)
- Scheduled imports (batch upload schedule)
Files to Create/Modify
Create:
app/(protected)/admin/bulk-import/page.tsx— admin pagecomponents/admin/bulk-import-form.tsx— upload + preview componentlib/incidents/bulk-import.ts— metadata extraction + incident factoryapp/api/admin/bulk-import/route.ts— upload handlerapp/api/admin/bulk-import/create/route.ts— batch creation endpoint
Modify:
app/(protected)/admin/page.tsx— add link to bulk importlib/storage/evidence.ts— no changes (use existinguploadEvidenceFile())
Optional:
supabase/migrations/####_bulk_import_sessions.sql— audit table (if needed)