Case study
PROVA
Upload model risk documentation, get an SR 11-7 compliance score and a list of what's missing.
- recall, 60 of 60 must-detect gaps
- 1.00
- precision on flagged gaps
- 0.77
- severity agreement with labels
- 0.70
The problem
Banks have to review model documentation against SR 11-7, the Fed's model risk guidance. It's slow, manual, and inconsistent between reviewers. PROVA does a first pass: a score, the gaps, and the evidence for each one.
How it works
- 01
Upload
The file is parsed in memory and the buffer is zeroed. Only the extracted text is saved, so you can revisit a submission.
- 02
Assess
Three Claude Haiku 4.5 agents run in parallel, one per pillar: Conceptual Soundness (7 elements), Outcomes Analysis (7), Ongoing Monitoring (6). Outputs are schema-checked with Zod.
- 03
Judge
A judge agent reviews all three outputs for consistency, taking a deliberately contrarian stance.
- 04
Score
Computed in code, not by the model: 100 − 20 per critical gap, 10 per major, 5 per minor.
- 05
Report
Per-element gaps with cited evidence, and a downloadable PDF.
Results
Scored against 8 synthetic documents × 20 elements. Claude wrote the documents and drafted the labels, and a Claude model does the grading, so this is partly circular. Treat it as a sanity check until a person relabels a sample.
Recall of 1.00 on 8 documents is the number I trust least. Precision is the more reliable number.
Decisions and dead ends
Don't let the model do the math
The agents' own scores didn't add up to their own findings. So the model only reports gaps and severities, and the score is a subtraction in code.
Contrarian judge
LLM judges tend to agree with themselves and favor whatever comes first. The judge is prompted to take the contrarian side, which counters self-enhancement and position bias.
Untrusted input
Document text sits inside XML delimiters and is sanitized server-side, so instructions buried in a document are read as content, not commands.
Regression runs against the live model
A weekly run scores 7 fixture documents with the real model and flags any final score that moves more than 10 points. PRs run lint, types, unit and E2E tests, secret scanning, and a build.
Per-user data isolation
Row-level security isolates each user's data. Benchmarks pool scores across users, so medians stay hidden until a model type has at least 5 submissions.
Stack
TypeScript · Next.js · Claude Haiku 4.5 · Zod · Supabase · PostgreSQL · Jest · Playwright · GitHub Actions · Vercel · Sentry