Observability Case Study
See inside every call.
A full observability layer over a production RAG system: every query traced into retrieval, reranking, and generation time, with Grafana dashboards and CI gates that block a deploy before it degrades quality.
Production RAG
Langfuse · Prometheus · Grafana
2025
p50 / p95 latency
token cost / query
CI regression gate
412msp50 latency
891msp95 latency
$0.006cost / query
0.94quality score
01 — The problem
Debugging a RAG pipeline blind.
✕Which stage is slow?
✕Token spend was invisible
✕Quality drops shipped silently
Attribution
A single end-to-end latency number doesn't say whether retrieval, reranking, or generation caused a slowdown.
Cost
Without per-query cost tracking, a prompt or model change could quietly inflate spend before anyone noticed.
Regression
Nothing stopped a change that degraded answer quality from reaching production before a human noticed.
02 — The approach
How a query gets observed.
01 Tracing
End-to-end spans
Langfuse instruments every request into retrieval, reranking, and generation spans with their own timing.
02 Metrics
Prometheus export
Latency, cost, and quality metrics scraped from the FastAPI service on every request.
03 Dashboards
Grafana
Tracks p50/p95 latency, token cost per query, and quality scores over time.
04 Gate
Regression check
A GitHub Actions check blocks a deploy when latency spikes or a quality score drops.
03 — Under the hood
Four engineering decisions, up close.
Tracing
Span-level, not request-level.
Langfuse spans wrap each pipeline stage individually, so a latency spike can be attributed to retrieval versus generation instead of guessed at.
span: retrieval → 140ms
span: rerank → 90ms
span: generation → 660ms
Dashboards
One place for latency, cost, and quality.
Grafana panels pull from the same Prometheus metrics store, so latency percentiles, token cost, and quality scores live side by side.
panel: latency_p50 / p95
panel: cost_per_query
panel: quality_score
CI gating
The pipeline fails closed.
A GitHub Actions job runs the evaluation suite on every change; a regression in latency or quality score fails the check instead of shipping silently.
if quality_score < threshold:
fail_build()
Serving
Instrumented at the FastAPI layer.
Metrics are emitted directly from the serving layer, so the dashboards reflect real production traffic, not a synthetic sample.
middleware: metrics_emit
source: live production traffic
04 — Results
What became visible.
Impact
Pipeline stages traced individually
3 stages
p50 / p95
Latency percentiles tracked continuously on Grafana, not just averages.
Per-query
Token cost tracked at the individual query level, not just in aggregate.
Auto-blocked
CI gating stops a deploy when latency or quality regresses.
Live traffic
Dashboards reflect real production requests, not a synthetic sample.
05 — Honest scope
What this layer doesn't do yet.
No alerting yet
Dashboards surface regressions visually; there's no paging or Slack alert wired up for on-call response.
Single-service scope
Tracing covers the RAG service itself, not every upstream or downstream dependency.
Fixed evaluation thresholds
The CI quality gate uses static thresholds rather than statistical drift detection.
Alerting
Wire Grafana alert rules into Slack or PagerDuty for real-time incident response.
Drift detection
Replace static thresholds with statistical drift detection on quality scores over time.