Fintech
Document intelligence
A grounded extraction-and-review pipeline for dense financial filings: it reads every document, extracts the fields downstream systems need, gates low-confidence output to a human, and routes the rest straight through — with every answer traced to its source span and measured against an eval suite.
- LangGraph
- pgvector
- Postgres
- Go
manual review
throughput per analyst
In short
- Analysts stopped reading every filing — only low-confidence extractions reach a human now.
- Every extracted field traces back to the exact source span it came from.
- Accuracy is measured on a labelled eval suite that runs in CI, not assumed.
The engagement
- Sector
- Financial services (B2B)
- Company shape
- Series B, ~200 people
- Engagement
- Discovery → build → handover, ~14 weeks
- Team
- 2 engineers + a fractional ML lead
- Region
- North America
Anonymised by policy
The challenge
Every filing landed as a dense, inconsistently-formatted PDF, and an analyst had to read the whole thing before any downstream process could start. Volume grew faster than the team could hire, review times stretched into days, and the manual step was the bottleneck for the entire product.
Off-the-shelf OCR and a naive LLM prompt both failed the bar that mattered: finance can't act on a plausible-looking number that isn't traceable to the source. Any system had to be right, cite where it got each value, and know when it wasn't sure.
The approach
- 1Ground the problem in a real eval set
We built a labelled evaluation set of representative filings first, so every later change was measured, not guessed. Accuracy targets and a confidence threshold were agreed up front.
- 2Retrieval over generation
Documents are chunked and embedded; extraction is grounded in retrieved spans rather than free generation, so every field maps back to the exact source text.
- 3Confidence gating, not blind automation
A calibrated confidence score routes low-certainty extractions to a human review queue and lets high-certainty ones flow straight through — the human effort goes only where it changes the answer.
- 4Trace + observe everything
Each run is traced end-to-end (input → retrieved spans → extraction → decision), so a wrong answer is debuggable and the eval suite runs in CI on every change.
The solution
The pipeline ingests a filing, chunks and embeds it, and runs grounded extraction against the retrieved spans. A confidence gate splits the output: anything below the threshold is queued for a human with the source span pre-highlighted; everything above flows straight to structured output. The eval suite runs on every change, so a regression is caught before it ships — reliability is a measured property of the system, not a hope.
The system we built
- IngestPDF → normalized text
- Chunk + embedpgvector store
- Grounded extractionretrieval-first, cited spans
- Confidence gatethreshold routing
- Human reviewlow-confidence only
- Structured outputdownstream-ready
The results
manual review
vs. the fully-manual baseline
throughput per analyst
vs. the pre-automation baseline
hallucinated citations in eval
on the held-out eval set
Data & safety
- Evaluation
- A labelled eval suite runs in CI — extraction is scored on every change, not spot-checked.
- Guardrails
- A calibrated confidence gate routes low-certainty output to human review; only high-confidence fields flow straight through.
- Data handling
- Runs inside the client's environment — filings and extracted data never leave their systems.
- Traceability
- Every run is logged end-to-end (input → retrieved spans → decision) for audit.
The stack
- Orchestration
- LangGraph, Go
- Retrieval
- pgvector, Postgres
- Eval + tracing
- Custom eval suite, OpenTelemetry
The system tells us when it isn't sure — that's the part that let us trust it in production.