Multi-Agent Engineering Case Study
Never fabricate
an outcome.
Nimbus Support Agent is a LangGraph-orchestrated multi-agent support system, built phase by phase — from a no-persistence RAG bot to a system with real Stripe refunds, real Shopify order lookups, real Zendesk ticketing, and a live ops dashboard, all independently verified against the actual downstream APIs.
4 phases shipped
LangGraph · FastAPI · Supabase
2025
Built for Nimbus, a fictional smart-home brand. Stripe, Shopify, and Zendesk integrations run against real test-mode / sandboxed accounts — not live customer data.
gate: interrupt() if escalated
escalation check
response_composer only
This is the third time I've asked — I want a refund now.
triage → refund · sentiment: negative (3rd turn)
refund $85 ≥ $50 limit → escalation, not auto-approved
I've flagged this for a specialist to review right away — you'll hear back shortly, and I've logged the full context so you won't have to repeat it.
01 — The brief
Resolve the boring 80% — or say so honestly.
Nimbus is a fictional smart-home hardware company with a subscription add-on. The brief: build a support agent that resolves FAQ, order status, refunds, and billing end to end against real systems, while knowing precisely when to stop and hand off to a human.
✕Confidently wrong erodes trust
✕A prompt is a suggestion, not a guarantee
✕Nothing proves the outcome happened
The non-negotiable
An agent that answers confidently and is sometimes wrong is worse than useless in support. The whole build was organized around one rule: never fabricate an outcome.
Enforced in code
If the knowledge base doesn't cover a question, say so. If an order can't be found, say so. If a refund needs judgment, don't issue it — enforced in code, not requested in a prompt.
Four phases
Reasoning (Phase 1), safety & memory (Phase 2), real-money & real-inventory integrations (Phase 3), observability for humans in the loop (Phase 4).
02 — Architecture
A typed state machine, not a prompt loop.
The core is a typed LangGraph state graph. Every turn moves through explicit, named nodes with conditional edges — only one node is ever allowed to write the customer-facing reply.
01 Gate
Pause if escalated
Pauses via LangGraph's interrupt() if the conversation is already escalated, so the bot can't respond over a human's head.
02 Triage
Route by intent
gpt-4o-mini classifies intent while sentiment scores the same message in parallel, same superstep.
03 Specialists
faq_rag · refund · order_status
Four independent nodes, including an honest unsupported_intent placeholder rather than a fake resolution.
04 Escalation check
The only branch point
Every specialist's result passes through one shared check that decides human handoff.
05 response_composer
Sole writer of the reply
The only node permitted to write the customer-facing message — tone and honesty enforced in exactly one place.
Schema written once: every cross-phase field (sentiment_score, escalation_reason...) existed from Phase 1 onward and sat inert until needed — Phases 2–4 never required a breaking migration.
Key decision
Guardrails as code, not prompt.
The refund auto-approval limit is an if amount >= limit branch that never reaches the Stripe call — not an instruction the model could be talked out of. Tradeoff: every new policy needs a code change, not a prompt edit.
if amount >= limit:
escalate() # never reaches Stripe
Key decision
Honest "I don't know" over confident guessing.
RAG returns a NO_RELEVANT_DOCS sentinel below a similarity threshold instead of answering from a weak match; a missing order number escalates instead of assuming which order is meant.
similarity < threshold →
NO_RELEVANT_DOCS
composer → "I don't have that"
Key decision
Shopify token scoped read-only.
A support bot never needs to write an order. write_orders was deliberately never requested, so an order-mutation bug is architecturally impossible, not just untested.
scope: read_orders, read_customers
write_orders: never requested
Key decision
Best-effort persistence and ticketing.
Supabase writes and Zendesk ticket creation are caught and logged, never raised — an outage in either system degrades to a missing ticket ID, not a broken customer reply.
try: create_ticket()
except: log_and_continue()
03 — Build, phase by phase
Four shipped phases, one axis of capability each.
Rather than widening everything at once, each phase added exactly one axis: reasoning, then safety and memory, then real integrations, then observability.
Phase 1 — MVP
Triage → grounded RAG → response, no persistence
The smallest end-to-end loop: gpt-4o-mini classifies intent, a Chroma-backed RAG node answers strictly from retrieved knowledge-base docs, and a composer writes the reply. Every non-FAQ intent routed to an honest placeholder rather than pretending a refund or order lookup had happened.
VerifiedIn-KB questions answered with real policy numbers; out-of-KB questions got an honest non-answer instead of a hallucination; an angry refund request on an unlooked-up order never fabricated a result.
Phase 2 — Sentiment, escalation, memory
The safety net and the human-in-the-loop pause
This is where it stopped being a chatbot and became a support system: gate pauses via interrupt() whenever a conversation is already escalated, sentiment tracks in parallel with triage, five escalation triggers, and Supabase persistence with cross-conversation memory.
VerifiedRefunds branch correctly at the $50 limit with matching database rows; three turns of worsening sentiment escalate on exactly the turn that crosses the threshold; an escalated conversation genuinely pauses and resumes only through the resolve action.
Phase 3 — Real money, real inventory
Live Stripe refunds, live Shopify order status
Phase 2's fake stubs were replaced with real test-mode integrations. The auto-approval limit became a hard branch in the refund node; Shopify order lookups run against a real dev-store Admin API with a deliberately read-only token. order_not_found joined the escalation triggers.
VerifiedA refund under the limit was independently confirmed via the Stripe API as actually issued; a refund at the limit was confirmed as not issued — the guardrail held at the API level, not just in the reply text.
Phase 4 — Dashboard & real ticketing
A live ops surface for the humans in the loop
A Next.js 16 dashboard behind Supabase Auth: a live conversation viewer on Supabase Realtime, an escalation queue with a working resolve action, and sentiment analytics. Zendesk ticket creation went live via OAuth client_credentials.
VerifiedReal HTTP requests confirmed auth gating, live counts matching actual row counts, and the resolve button correctly clearing escalations for both a genuinely paused conversation and the far more common case of one that was never paused at all.
In progress — Deployment hardening
Postgres-backed checkpointer, Fly.io deploy
Swapping the local SQLite checkpointer for PostgresSaver against Supabase when SUPABASE_DB_URL is set, falling back to SQLite for zero-setup local dev. A Dockerfile and fly.toml target Fly.io with a persistent volume for the Chroma index.
04 — Bugs found in verification
Caught by querying real systems, not green tests.
Every phase ended with acceptance checks against actual Stripe, Supabase, and Zendesk state — not unit tests confirming the code compiles. Five bugs stood out.
01 Chroma's default distance metric silently broke retrieval
Chroma's default L2-distance-to-relevance conversion isn't calibrated for OpenAI embeddings — it produced near-random scores, so the FAQ node found zero relevant docs even for questions squarely in the knowledge base.
Fix: switched the collection to cosine distance and computed similarity as 1 − distance, recalibrating the threshold.
02 A resume message corrupted the shared transcript
The gate node injected a synthetic "[Human agent resolved…]" message after a resume. Every node assumes messages[-1] is the real customer turn — and the injected text matched the escalation-request regex, re-escalating the conversation immediately after it was resolved.
Fix: the human's note is logged, never written into the shared transcript.
03 Stripe's SDK object model wasn't actually a dict
StripeObject implements [] and in but not .get() or .keys() — dict(charge.metadata) also failed, crashing with a KeyError on a numeric index.
Fix: switched to "order_id" in charge.metadata. Logged as the reason the project later chose plain REST over the Zendesk SDK.
04 A shared session helper tore down state a caller still needed
Shopify's find_order_by_number manages its own session lifecycle — but the seed script called it, and its internal clear_session() tore down the session the seed script needed next, raising ValueError: No shopify session is active.
Fix: the seed script re-activates its own session immediately after any such lookup.
05 Resolving an escalation was a silent no-op most of the time
The resolve endpoint unconditionally called Command(resume=...) — which only does anything if that conversation is actually paused. Most escalations happen with no follow-up message ever sent, so the call returned a normal-looking response but never touched escalations.resolved_at.
Fix: check get_state(config).next first; if nothing is paused, clear the escalation directly.
05 — Results
What "verified" actually meant.
A refund "issued" was confirmed by calling the Stripe API directly and checking charge.refunded. A Zendesk ticket was fetched back and compared against the real escalation reason. A knowledge-base edit was confirmed by re-asking the question — then confirming the old answer returned after a delete.
Impact
Bugs caught by testing against real systems, not a green test run
5 real bugs
4 phases
Shipped from a no-persistence MVP to a live ops dashboard.
3 integrations
Stripe, Shopify, and Zendesk — real test-mode APIs, not mocks.
6 triggers
Independent escalation triggers, so no single failure mode has to catch everything.
0 migrations
The state schema was written once — no breaking migration across 4 phases.
LangGraphgpt-4o / gpt-4o-miniLangChain + ChromaPython · FastAPISupabase PostgresNext.js 16Stripe (test mode)Shopify Admin APIZendesk REST · OAuthFly.io (in progress)
06 — Honest scope
What this doesn't do yet — on purpose.
All four planned phases are complete and independently verified. Two extensions shipped past the original spec: billing_question now shares the faq_rag specialist, and the knowledge base moved from static markdown files to a real dashboard-editable admin surface backed by Postgres.
Deployment hardening in flight
The Postgres-backed checkpointer and Fly.io deployment are needed so conversation state survives across machines and deploys, not just process restarts.
No voice channel or multi-language support
Deliberately out of scope from the original design — each would trade a currently-auditable guardrail for a harder-to-verify one.
No learned refund policy
The refund limit stays a fixed, auditable branch rather than a model-learned policy, for the same reason.
Storefront widget
A Shopify storefront extension is scaffolded as the likely next customer-facing surface, beyond the current CLI/dashboard access points.
Multi-machine durability
Swap the SQLite checkpointer for Postgres so state survives a real multi-machine deploy, not just a restart.