Case study
RAG-SEC
Ask a question about any of 799 SEC 10-K filings and get an answer with citations.
- recall@10 on 1,545 T²-RAGBench test questions
- 0.831
- refusals on a hand-built unanswerable set
- 47 / 47
- median end to end, 10 questions on a T4 GPU
- 6.6 s
- per answer at standard prices, model plus GPU
- $0.013
The problem
Questions about 10-Ks usually need one company, one year, often one table. Generic retrieval returns boilerplate that looks relevant and isn't. Each retrieval step here stayed only if it raised recall.
How it works
- 01
Query
Detect the company, restrict search to its filings, strip boilerplate from the question.
- 02
Retrieve
BGE-M3 dense search (pgvector, HNSW) and BM25 (pg_search), plus a list of chunks that name the filing year.
- 03
Fuse
Reciprocal rank fusion (k = 60) over the three lists, with a small boost for the asked-about year. Top 50 kept.
- 04
Rerank
bge-reranker-v2-m3 cross-encoder narrows 50 to 10.
- 05
Answer
Gemini 3.7 Flash writes a cited answer. The prompt tells it to answer only from the evidence and refuse otherwise.
Results
Settings were chosen on dev. Test was scored alongside dev before each change shipped, so it's a reused hold-out and 0.831 may be slightly optimistic.
| Stage | Dev | Test |
|---|---|---|
| Dense search only | 0.337 | – |
| + BM25, fused with RRF | 0.514 | – |
| + cross-encoder reranker | 0.629 | 0.607 |
| + company filter, name removed from rerank query | 0.760 | 0.747 |
| + filing-year boost in RRF | 0.791 | 0.771 |
| + name removed from dense query | 0.833 | 0.814 |
| + year-mention list, read 200 deep | 0.847 | 0.831 |
The company filter and the name strip only work together. On dev, the filter alone gave +0.031 and the strip alone +0.010. Together, +0.126. Once you only search one company's filings, every chunk mentions that company, so the name in the query just rewards generic boilerplate.
Decisions and dead ends
Tried an agent. Didn't ship it.
A LangGraph plan, retrieve, judge, retry loop first beat one-shot 14 to 1 on disagreeing questions (p = 0.001). Against a fair baseline that shrank to 8 to 2: 74.0% vs 70.5% on 200 dev questions, McNemar p = 0.109. Not shown to help, and slower, so it went out as a negative result.
Work moved upstream of the reranker
After adding the reranker, dev recall@10 was already 89% of recall@50 (0.629 / 0.708). The reranker was sorting well. The misses were chunks that never made the top 50, so the later work went into query handling and fusion.
Checked the labels before trusting the score
A matcher pinned the gold lines for 95.1% of the 11,739 T²-RAGBench questions in the corpus; the rest fall back to page-level matching. A blind audit of 660 cases by 15 parallel LLM review agents found label noise wasn't correlated with failures.
Cost and latency
One evidence block was 89% of spend, and hidden reasoning tokens were about 85% of billed output. The full dev and test benchmark cost $14.69 on batch inference. On a T4 GPU, 3.3 s of the 3.6 s median retrieval time was the cross-encoder. Without the GPU, as in the demo, reranking takes about 26 s. That's the next thing to cut.
CI guards the number
CI replays stored rerank scores for all 1,545 test questions and fails unless recall@10 is exactly 0.831. No GPU, database, or model calls, so it's cheap to run every time.
Stack
Python · PostgreSQL · pgvector · pg_search · Hugging Face · Gemini API · LangGraph · Langfuse · FastAPI · Docker · AWS EC2 · GitHub Actions