AI System, Prompts, Citations, Evals, and Approval Queue

This guide ties together the CaseSolo AI modules and the docs that already cover individual subsystems.

Core Modules

• Provider routing: src/lib/ai/provider.ts
• Prompt library: src/lib/ai/prompt-library.ts
• Execution envelope: src/lib/ai/execute.ts
• Output envelope: src/lib/ai/output-envelope.ts
• Citations: src/lib/ai/citation.ts
• RAG pipeline: src/lib/ai/rag-pipeline.ts
• Retrieval and embeddings: src/lib/ai/retrieval.ts, src/lib/ai/embeddings.ts
• Review queue: src/lib/ai/review-queue.ts
• Evals: src/lib/ai/evals.ts
• Governance metrics: src/lib/ai/governance-metrics.ts
• Truth Guard: src/lib/ai/truth-guard.ts

Related docs:

• AI Output Envelope
• AI Review Queue
• AI Swarm
• AI Evals
• AI Governance Metrics
• RAG Pipeline
• Truth Guard

Prompt Standards

Prompts must be versioned and testable. A prompt change should identify:

• Prompt ID and semantic version.
• Owning workflow.
• Input schema and required context.
• Output schema or envelope fields.
• Citation requirement.
• Approval requirement.
• Eval fixtures affected by the change.

Avoid embedding firm secrets, raw OAuth tokens, or private signed URLs in model input. Use document IDs, storage keys, citations, or short excerpts instead.

Citation Rules

Legal, medical, settlement, deadline, and case-strategy outputs should cite their source material. Citation payloads should include:

• Source type, such as document, case note, statute rule, or knowledge article.
• Source ID or stable locator.
• Snippet or page span when available.
• Confidence and any missing-source warning.

If a task requires citations and the system cannot find usable source material, the AI should refuse the cited claim or send the result to review.

Approval Queue

Generated legal work that mutates case state or leaves the system should pass through src/lib/ai/review-queue.ts unless a deterministic route explicitly marks the workflow as safe.

Use approval queue records for:

• Demand letters, pleadings, releases, and settlement communications.
• Client-facing portal messages generated by AI.
• Medical chronology or liability summaries used as work product.
• Workflow automation changes proposed by AI.
• Any output with low confidence, missing citations, or policy warnings.

Approval records should preserve prompt version, model, tools, citations, cost, latency, actor, firm ID, case ID, and reviewer decision.

Evals

Evals should be deterministic and offline-friendly. Add fixtures when changing:

• Prompt behavior.
• Citation enforcement.
• Model routing.
• Truth Guard rules.
• Review queue routing.
• RAG chunking or retrieval behavior.

Run focused eval tests with:

npm run ai:evals

For launch-sensitive changes, also run:

npm run test:run -- src/__tests__/lib/ai-evals.test.ts

Governance Metrics

AI production readiness depends on measurable controls:

• Approval rate and correction rate.
• Citation coverage.
• Hallucination or unsupported-claim reports.
• Prompt version adoption.
• Cost and latency by workflow.
• Provider readiness and fallback behavior.

Do not treat a new AI workflow as launch-ready until it has a prompt owner, focused tests or evals, approval policy, and metric visibility.