Document Intelligence Pipeline

The document intelligence pipeline turns OCR text into structured case metadata for search, review, billing, and medical workflows.

Processing Path

Uploads can enter two OCR paths:

• src/lib/documents/ocr-worker.ts handles dedicated DOCUMENT_OCR jobs.
• src/lib/jobs/processors/document-processing.ts handles the full document-processing job and now calls the same higher-level OCR pipeline before RAG ingestion.

Both paths use processDocumentOcr() from src/lib/documents/ocr-pipeline.ts.

OCR Strategy

The pipeline selects providers in this order:

• PDF documents: native text extraction with pdf-parse, then AWS Textract when configured, then local Tesseract fallback.
• Images: AWS Textract when configured, then local Tesseract fallback.
• Plain text: direct UTF-8 text extraction.

Textract is enabled only when AWS Textract credentials are present. Missing or failing cloud OCR does not block the local fallback.

Structured Extraction

After text extraction, src/lib/documents/document-intelligence.ts derives:

• ICD-10 diagnosis codes.
• CPT/procedure codes, only when billing/procedure context is nearby.
• Provider candidates, nearby NPI numbers, and inferred specialty hints.
• Billing amounts, including total billed, paid, balance due, and patient responsibility.
• PII findings for SSNs, dates of birth, phone numbers, emails, and medical record numbers.
• Searchable PDF readiness, including whether a PDF already has a native text layer or needs a searchable derivative.

The extractor is deterministic and bounded to the first 80,000 characters to keep worker latency predictable.

Persisted Metadata

Successful OCR stores these fields on Document.metadata:

• extractedText
• extractedEntities
• documentIntelligence
• icd10Codes
• cptCodes
• providerCandidates
• billingExtraction
• piiDetection
• searchablePdf
• ocrTextPreview
• ocrPageCount, ocrWordCount, ocrConfidence
• classifiedType, classificationConfidence, classificationMetadata

GET /api/documents/[documentId]/ocr exposes the preview and intelligence fields so UI surfaces do not need to fetch the full OCR body for summaries.

Verification

Run the focused deterministic checks with:

npm run test:run -- document-intelligence

For task-level verification, also run targeted ESLint and npm run typecheck.