Files
ims/docs/superpowers/specs/2026-08-16-bulk-incident-import-design.md
T

8.9 KiB

Bulk Incident Import — Design Specification

Date: 2026-08-16
Status: Ready for Implementation
Scope: Admin-only bulk import of historical incident cases with AI-extracted metadata


Overview

Problem: Hundreds of historical incident case folders (containing PDFs, images, videos) need to be migrated into IMS as closed incidents for record-keeping and archival.

Solution: Bulk import system that accepts single-folder or .zip-based uploads, analyzes PDFs via DeepSeek API to extract incident metadata, and creates closed incidents with attached evidence files.

Success Criteria:

  • Admin can upload single folder or .zip containing multiple case folders
  • System auto-extracts metadata from PDFs (description, date, location, type, severity, witnesses)
  • Admin reviews extracted data, fills required gaps, approves batch
  • All incidents created as "Closed" status with full evidence file chain
  • Audit log captures bulk import source for each incident

Architecture

Components

1. Upload Handler (POST /api/admin/bulk-import)

  • Accept multipart/form-data with single folder upload OR .zip file
  • Extract files, group by folder (each folder = one incident)
  • Validate: at least one file per folder, PDFs present for analysis
  • Response: list of folders ready for metadata extraction

2. PDF Metadata Analyzer

  • Read PDF/DOC content (text extraction via file-type detection)
  • Send text to DeepSeek API with structured extraction prompt
  • Extract fields: description, date, location (site + zone), incident_type, priority, witnesses
  • Return structured JSON for each folder's PDFs
  • Cache results during session (don't re-analyze on retry)

DeepSeek Prompt Template:

Analyze the following incident document(s) and extract:
- Description: summary of what happened
- Date: incident date (YYYY-MM-DD format)
- Location: site name and zone/area
- Type: incident category (env/asset/security/fire)
- Priority: severity level (low/med/high)
- Witnesses: names and contact info if present

Document text:
[PDF content here]

Return JSON format:
{
  "description": "...",
  "date": "YYYY-MM-DD",
  "location": {
    "site": "...",
    "zone": "..."
  },
  "incident_type": "env|asset|security|fire",
  "priority": "low|med|high",
  "witnesses": [{"name": "...", "contact": "..."}, ...]
}

3. Preview & Approval Table

  • Display extracted data as editable table
  • Columns: folder name, description, date, location (site), location (zone), type, priority, # witnesses, # files
  • Allow inline editing for any field (admin fills gaps)
  • Mark rows as complete/incomplete (red if required field missing)
  • "Import All" button only enabled when all rows have description + date + location

4. Incident Factory

  • Create incident from row data using withUser() transaction
  • Set status = "Closed" (not "Open")
  • Set created_by = current admin user
  • Store bulk_import_source in audit_log old_value for tracking
  • Attach all folder files as evidence (PDFs, images, videos)
  • Generate unique incident reference_no (existing format: SITE-YYYYMM-####)

5. Evidence Attachment

  • For each file in folder: call uploadEvidenceFile() (existing helper)
  • Store in /var/lib/ims-evidence/[incident_id]/[filename]
  • Record in evidence_files table (same as regular incidents)
  • No special staging/approval; all files attached as "verified" evidence

Data Model

New Table: bulk_import_sessions (optional, for audit)

CREATE TABLE bulk_import_sessions (
  id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
  imported_by UUID NOT NULL REFERENCES users(id),
  upload_filename TEXT NOT NULL,
  folder_count INT NOT NULL,
  incident_count INT NOT NULL,
  file_count INT NOT NULL,
  created_at TIMESTAMP DEFAULT now()
);

Existing Tables Modified

  • incidents: use existing columns (description, created_by, status, etc.)
  • evidence_files: attach files under incident_id (existing behavior)
  • audit_log: new entries for each bulk-imported incident with action='bulk_import'

User Flow

  1. Upload: Admin navigates to /admin/bulk-import, uploads single folder or .zip
  2. Analysis: System extracts files, analyzes PDFs via DeepSeek, shows progress
  3. Preview: Table displays extracted metadata; admin reviews and fills gaps
  4. Approval: Admin clicks "Import All" (button disabled if any required field missing)
  5. Creation: System creates all incidents + attaches files in parallel
  6. Confirmation: Shows summary (X incidents created, Y total files imported)

API Routes

POST /api/admin/bulk-import — Upload & Analyze

Request:

multipart/form-data:
  file: <single-folder.zip or folder>
  (or individual folder via file picker)

Response:

{
  "status": "success",
  "folders": [
    {
      "id": "folder_hash",
      "folder_name": "10AUG2026 - SUNIL",
      "extracted": {
        "description": "...",
        "date": "2026-08-10",
        "location": { "site": "Main WH", "zone": "Zone 7" },
        "incident_type": "asset",
        "priority": "high",
        "witnesses": [...]
      },
      "files": {
        "pdfs": 2,
        "images": 3,
        "videos": 1
      },
      "metadata_confidence": 0.92
    }
  ]
}

POST /api/admin/bulk-import/create — Batch Create

Request:

{
  "folders": [
    {
      "id": "folder_hash",
      "description": "...",
      "date": "2026-08-10",
      "location": { "site": "...", "zone": "..." },
      "incident_type": "asset",
      "priority": "high",
      "witnesses": [...]
    }
  ]
}

Response:

{
  "status": "success",
  "created_incidents": [
    {
      "id": "inc-123",
      "reference_no": "MAIN-202608-0001",
      "description": "...",
      "status": "closed",
      "file_count": 6
    }
  ],
  "total_files_imported": 42
}

UI Page: /admin/bulk-import

Layout:

  • Header: "Bulk Import Historical Incidents"
  • Upload section: folder/zip picker + drag-drop zone
  • Progress bar during analysis
  • Preview table (editable)
  • "Import All" button + confirmation modal
  • Success summary with incident list

Permissions: Admin role only (requireAdmin())


Error Handling

Scenario Behavior
No PDFs in folder Skip folder, warn admin ("No analyzable documents in 'folder_name'")
DeepSeek API fails Show error, allow manual entry (fields stay empty)
Missing required field after manual review Disable "Import" button, highlight row in red
File upload too large (>100MB total) Reject, suggest splitting into multiple uploads
Folder name conflicts with existing incident Auto-increment reference_no, notify admin
Database transaction fails on create Rollback entire batch, show error with failed folder names

Testing Strategy

Unit Tests:

  • PDF text extraction (mock file reads)
  • DeepSeek prompt generation and JSON parsing
  • Incident factory (create incident, attach evidence, audit log)
  • Reference_no generation under collision scenarios

Integration Tests:

  • End-to-end single-folder upload → preview → create
  • Batch .zip upload with 5+ folders
  • Missing required fields → error handling
  • Evidence file attachment verification

Manual Testing:

  • Upload real historical case folder
  • Verify extracted metadata accuracy
  • Confirm incident + evidence files appear in HSE incidents list
  • Check audit_log for bulk_import entries

Deployment Notes

Environment:

  • DEEPSEEK_API_KEY must be set (already configured for risk analysis)
  • EVIDENCE_DIR must exist and be writable (already set)
  • Supabase storage for temporary zip extraction (use /tmp or disk)

Performance:

  • DeepSeek analysis runs sequentially per folder (safe for hundreds)
  • Incident creation (batch) can run in parallel to save time
  • Estimated time: ~5-10 minutes for 100 folders (depends on PDF sizes + DeepSeek latency)

Compliance:

  • Audit log tracks every bulk import (action='bulk_import', old_value=session_id)
  • Closed status prevents accidental edits (no workflow triggers)
  • Evidence retention policy applies (5-year minimum per DOSH)

Future Enhancements (out of scope)

  • Async job queue for very large imports (1000+ folders)
  • OCR preprocessing for scanned PDFs
  • Duplicate detection (warn if folder name matches existing incident)
  • Scheduled imports (batch upload schedule)

Files to Create/Modify

Create:

  • app/(protected)/admin/bulk-import/page.tsx — admin page
  • components/admin/bulk-import-form.tsx — upload + preview component
  • lib/incidents/bulk-import.ts — metadata extraction + incident factory
  • app/api/admin/bulk-import/route.ts — upload handler
  • app/api/admin/bulk-import/create/route.ts — batch creation endpoint

Modify:

  • app/(protected)/admin/page.tsx — add link to bulk import
  • lib/storage/evidence.ts — no changes (use existing uploadEvidenceFile())

Optional:

  • supabase/migrations/####_bulk_import_sessions.sql — audit table (if needed)