Solo Engineering Case Study
Five days, from empty repo
to a self-grading retrieval system.
Financial RAG Engine is a production-shaped Retrieval-Augmented Generation service for 10-K filings and earnings reports — hybrid dense/sparse retrieval, contextual chunk enrichment, a Self-RAG feedback loop, and a fail-safe integration pattern that keeps the whole thing running even when Redis or Langfuse are unreachable.
1 engineer · 5 days
11 commits · v2.0.0
FastAPI · AWS ECS
route: rag
RRF fusion, k=60
Self-RAG: sufficient
What was NVIDIA's operating margin in the segment discussed on the prior page?
cache: miss (resolved query, cosine < 0.92)
hybrid: Pinecone + BM25 → rerank → grade: sufficient
Operating margin for the Data Center segment was 68% [1], up from the prior quarter's figure discussed in the same filing [2].
01 — Overview
Answer with citations, without inventing numbers.
Financial analysts read 10-Ks the way everyone dreads reading 10-Ks: skimming a 200-page PDF for the three sentences that matter, hoping keyword search doesn't miss the one that's phrased differently than expected. The premise is narrow on purpose — take a single financial filing, and answer questions about it with citations, without inventing numbers.
✕200-page PDFs waste analyst time
✕Keyword search misses rephrasing
✕Invented numbers are unacceptable
What makes it worth a case study isn't the premise, it's the shape of the engineering: a solo build, done in five consecutive days, that carries the instrumentation, fail-safes, and test discipline of a system meant to survive contact with production — deployed continuously to AWS ECS behind a GitHub Actions pipeline from day one.
02 — Build timeline
Eleven commits, four working days.
Reconstructed from git log. Infrastructure plumbing interleaved with product features, because the pipeline was live on ECS almost immediately.
Day 1 — May 1
7777332 Initial commit — the whole v1 system lands at once: 18 files, ~3,000 lines. FastAPI app, five services (document processor, embeddings, Pinecone, reranker, semantic cache), a vanilla JS frontend, a Ragas eval script, AWS/Docker deploy scaffolding already in place.
1ef41e0 Dependency fix — Dockerfile patched same evening, first contact with "works locally, fails in the container."
c60f82f AWS region corrected to line up with the real ECR/ECS account.
Day 2 — May 2
0b3eadb ECS task definition committed, deploy job wired up for real.
93ba30f 1aca126 Two follow-up commits tuning the task definition — the unglamorous part of shipping a container to a real cluster.
Day 3 — May 4
9af2d76 Added tests — a real suite arrives in one pass: pytest.ini, unit tests for the document processor and LLM service, an integration suite, a CI job gating deploys, a rewritten README with an architecture diagram.
74b6689 requirements.txt shrinks 4,756 → 2,247 bytes — unused dependencies trimmed after seeing what CI actually needed.
3bd6e63 733e647 Settings loosened to accept placeholder keys in CI so pytest can boot the app with every external call mocked.
Day 4 — May 5
8dd77cb Added inline citations — bracketed [1] [2] citations threaded from the reranker through the generator prompt into the frontend.
Working tree — today
Hybrid retrieval + tracing — three new services (bm25_service.py, hybrid_retriever.py, tracing.py) plus a rewritten /search pipeline and a new benchmark script sit uncommitted. See section 06.
03 — Architecture
Two pipelines: one writes, one reads.
Ingestion happens once per document and is allowed to be slow and thorough. Retrieval happens on every keystroke a user waits on, and is built to degrade rather than fail.
Ingestion — POST /ingest
01
PDF → Markdown
pymupdf4llm extracts text while preserving table structure.
02
Chunking
Recursive splitter, 1,000 chars / 200 overlap, paragraph-aware.
03
Enrichment
LLM writes a 1–2 sentence summary per chunk before embedding.
04
Embed
text-embedding-3-small → 1,536-dim vectors.
05
Dual index
Upsert to Pinecone (dense) and the in-memory BM25 index (sparse) together.
Retrieval — POST /search
0LLM
Route the query
Classify as rag / small_talk / off_topic before spending a retrieval call. Deterministic, 5-token cap.
1LLM
Resolve pronouns, expand
If "it / they / this…" and history exists, an LLM rewrites the entity, then generates 3 phrasing variants.
2Cache
Semantic cache lookup
Embed the resolved query, scan Redis for a cosine match ≥ 0.92. Hit → return in ~100ms.
3API
Embed variants
The 3 expansion variants are embedded (resolved query's embedding reused from the cache check).
4Local + API
Hybrid search
Dense (Pinecone) and sparse (BM25) run independently; fused with Reciprocal Rank Fusion, k = 60.
5Local
Cross-encoder rerank
FlashRank (ms-marco-MiniLM-L-12-v2, ONNX, CPU) cuts fused candidates to the top 5.
6LLM
Self-RAG grade
An LLM grader reads the top 3 of those 5 chunks and returns sufficient / insufficient.
7LLM
Rewrite & retry, once
If insufficient and a retry remains: rewrite with different vocabulary, re-embed, back to step 4.
8LLM
Generate the answer
Context-only system prompt, inline [1] [2] citations, history spliced in for follow-ups.
9Cache
Cache & trace
Store (resolved query embedding → answer) in Redis; flush the full Langfuse trace.
Schema note
Every ingestion stage is streamed through a StageTimer and reported to Langfuse as a span or generation.
04 — Engineering decisions
Six calls that shaped the system.
Each one trades something away on purpose — pulled directly from the codebase's own reasoning, not retrofit justifications.
Embeddings
text-embedding-3-small over a finance-tuned model
FinBERT and finance fine-tunes capture niche vocabulary better in isolation, but the constraint was latency and cost per query. A downstream cross-encoder rerank (FlashRank) recovers precision, so the cheap embedding only needs candidates in the top 10–20.
Trade-off: a slight edge lost on ultra-niche terms, for real-time latency and zero-shot generalization.
Chunking
Recursive splitting + Contextual Retrieval, not table-aware parsing
Tables split mid-row lose their headers — the classic RAG failure on 10-Ks. Instead of a specialized parser, an LLM reads a document prefix + preceding text and writes a summary restoring the table's meaning in text before embedding.
Trade-off: one extra LLM call per chunk, for never maintaining brittle table-parsing logic.
Retrieval
Hybrid dense + sparse, fused with RRF — not dense-only
Dense embeddings miss exact keywords (tickers, codes, figures); BM25 misses paraphrase. RRF combines both ranked lists without needing comparable scores. Both retrievers fail independently — zero hits on one falls back to the other.
Trade-off: a second retrieval path to maintain, for materially better recall on exact-term queries.
Caching
Cache the resolved query, at 0.92 similarity
Caching raw questions would let "how is it doing?" collide across conversations about different companies. The 0.92 threshold was tuned empirically — below 0.90, different questions matched; above 0.95, only near-identical phrasing hit.
Trade-off: queries under 6 words are never cached — too ambiguous to trust on similarity alone.
Self-RAG
One retry, then answer honestly with what's available
The grader gets one chance to reject context and force a rewrite. If still insufficient on the second attempt, the system doesn't error — it generates from the best context it has, explaining what's missing.
Trade-off: a bounded retry budget (max 2) caps latency, at the cost of occasionally giving up one attempt early.
Reliability
Every optional integration is fail-open, not fail-closed
SemanticCache and TracingService wrap their connection check in try/except and flip an _enabled flag false on failure — every public method becomes a silent no-op when disabled.
Trade-off: failures in these systems are silent by design — right for a cache/tracing layer, wrong for anything on the critical path.
05 — Evaluation
Measured with Ragas, not vibes.
A 5-question benchmark against NVIDIA's FY2025 10-K, scored on independent axes so a good answer can't hide a bad retrieval.
1.000
Faithfulness — every claim traces back to retrieved context.
Pass · target ≥ 0.95~0.930
Answer Relevancy — does the answer address the question asked.
Pass · target ≥ 0.901.000
Context Recall — were all needed chunks actually retrieved.
Pass · target ≥ 0.90—
Context Precision — how much of what was retrieved was noise.
Not yet reported
Honest read
Faithfulness and context recall at a perfect 1.0 on a 5-question set is a strong early signal, not yet a statistically wide one — "the approach works," not "the system is solved."
On the benchmark script
benchmark.py measures per-stage latency (p50/p95/p99 across the nine pipeline stages), fully wired to the hybrid pipeline but not yet run in this environment — no results file exists. Included as evidence that latency was treated as a first-class metric to measure, not just eyeballed end-to-end.
06 — Production & testing
Built to run unattended, not just to demo.
The deploy pipeline and hardening choices existed from the first commit — this was never a notebook with a FastAPI wrapper bolted on later.
Deploy path
GitHub Actions → ECR → ECS.
Build & push to ECR, render + deploy the ECS task definition, gated on the test job passing, triggered on push to main.
Containerpython:3.12-slim, layer-cached deps, FlashRank's ONNX model pre-baked at build time to avoid a cold-start download stall.
RuntimeUvicorn, 4 workers, health check at /health, static frontend served from the same FastAPI app.
Hardening on the request path
Six deliberate guardrails.
Streamed uploadPDFs read in 1MB chunks, hard 50MB cap enforced mid-stream — rejected before fully buffered into memory.
UUID filenamesEvery upload renamed to a UUID before touching disk — closes path traversal and collisions.
Per-IP rate limitsslowapi caps ingest at 5/min, search at 30/min, Ragas eval at 2/min.
Exponential backoffEvery OpenAI call retries up to 6 times, 1s → 32s with jitter, on RateLimitError.
Bounded concurrencyChunk contextualization caps at 3 concurrent LLM calls via a semaphore, 1s cooldown.
Known gap: CORS
allow_origins=["*"] — flagged in the code's own comment as "adjust in production."Testing
Arrived on day three, and it stuck.
Not part of the initial commit — landed as its own deliberate pass, and CI has gated every deploy since.
Unit — retrieval mathRRF fusion tested against the scoring formula: ordering preserved, shared docs boosted, dedup, empty-list edges.
Unit — LLM serviceRouting and small-talk generation tested with AsyncOpenAI fully mocked — no network calls, no key required.
IntegrationFastAPI's TestClient against /health, an empty-question 400, an invalid-file-type 400 on /ingest.
Coveragepytest-cov reports on every run; the --cov-fail-under=75 gate exists but is currently commented out — measured, not yet enforced.
Current state
What's sitting in the working tree right now.
The most architecturally significant change isn't in git history yet — the difference between the dense-only v1 pipeline and the hybrid v2 pipeline described in Architecture above.
main.pyLargest working-tree diff in the project's history (+212 / −62 lines) — /search rewritten around the Self-RAG retry loop and Langfuse instrumentation.
New filesbm25_service.py, hybrid_retriever.py, tracing.py — net-new and untracked, alongside a new hybrid-retrieval test file and benchmark.py.
07 — Honest scope
What holds up, and what a staff review would flag.
BM25's index lives entirely in RAM, rebuilt from Pinecone metadata on every cold start — fine now, but a full-corpus fetch loop won't stay cheap as document count grows.
The semantic cache does a linear scan + cosine comparison against every stored entry — correct, but O(n) against cache size rather than using Redis's vector search.
CORS is wide open by explicit, commented acknowledgment — the right call for a demo, the wrong one to forget before real traffic.
The 75%-coverage CI gate is written but commented out, so a coverage regression currently wouldn't block a merge.
The hybrid retrieval rewrite is unshipped — sitting in the working tree, not in a reviewed commit or a deployed image yet.
Fail-open design on every optional dependency means the core answer path never depends on infrastructure that wasn't provisioned.
Contextual Retrieval as a substitute for table-aware parsing turns a structural parsing problem into a text-generation problem the LLM already solves well.
Self-RAG's fallback avoids the two common failure modes: hallucinating past a gap, or hard-refusing when partial information exists.
Tests arrived before the pipeline got materially more complex, not after — RRF and BM25 unit tests were written alongside the feature.
The system doesn't try to be clever everywhere — it's dumb chunking plus a smart repair step, a cheap embedding plus a precise reranker, an optimistic cache plus a fail-open guard. Each layer compensates for the cheapness of the one below it.