The Four Subsystems
The reasoning core, the LLM itself, sits at the center. But the LLM alone has no continuity, no ability to act, no defenses, and no resilience. These four subsystems are what separate a production system from an impressive demo. Frameworks like LangGraph, Semantic Kernel, and CrewAI provide the scaffolding to wire them together. The scaffolding is not the architecture.
| Subsystem | What it does | What breaks without it |
|---|---|---|
| Memory | Gives the stateless model continuity and context | Agent forgets users, re-discovers the codebase every session, hallucinates history |
| Tool Use | Lets the agent interact with databases, APIs, and external systems | Agent can only produce text, no actions, no lookups, no real work |
| Guardrails | Rejects harmful inputs, enforces policies, filters dangerous outputs | Prompt injection, data leaks, regulatory violations, runaway costs |
| Fallbacks | Detects component failures and reroutes to degraded-but-functional alternatives | One failing API takes the entire agent offline |
Memory: The Part Everyone Gets Wrong
LLMs have no memory. Each API call arrives as a blank slate. The model that just helped you draft a contract has, by the next request, forgotten your name, the contract, and the conversation.
Production agents can't afford amnesia. A customer-service agent that forgets the complaint history isn't helpful. It's infuriating.
Production memory systems split into three types (borrowed from cognitive science):
Semantic memory stores factual knowledge. "This customer is on the Enterprise plan." "The deployment target is AWS us-east-1." It's the agent's reference library, doesn't change with each conversation, applies broadly.
Episodic memory records specific past events with temporal context. "On March 12, the user escalated a billing dispute." "Yesterday's deployment failed because of a missing environment variable." It gives the model a timeline, which is essential for reasoning about sequences and cause-and-effect.
Procedural memory captures how to do things. "When the user asks for a refund, check order status first, then verify the return window, then route to payments." Routines that shouldn't require re-thinking every time.
None of these live inside the model. They live in external systems: vector databases for semantic search, key-value stores like Redis for session context, relational databases for structured records, and specialized frameworks like Mem0, Zep, and Letta that abstract the plumbing.
The hard problem isn't storage. It's retrieval. We spent three weeks building a memory layer for a support agent, and on the first real test run it pulled in 14 "relevant" memories for a simple password reset request. The model got confused by a complaint from eight months ago about a different product and started apologizing for an issue the customer never mentioned. Too many memories drown the model in noise. Too few and it hallucinates confidently about things it should know.
The best systems treat memory retrieval as a search ranking problem: relevance scoring, recency weighting, and importance filtering, tuned to the task. This is sometimes called "context engineering," and it matters more than prompt engineering. The architectural challenge is retrieving the right memories at the right time without blowing past the context window (the maximum text the model can consider in a single call).
Tool Use: Where Most Production Agents Actually Break
A language model without tools can only produce text. Full of plans, unable to execute any of them. Tool use gives agents the ability to query databases, call APIs, send emails, execute searches, and interact with the systems where real work happens.
The Integration Problem
For the first generation of agents, tool integration was a nightmare. Every tool needed a custom connector with its own auth flow, error handling, and output parsing. Ten systems meant ten bespoke integrations. Teams reported spending 60-70% of their AI project time just building and maintaining connectors. This was the "N-times-M problem": N agents times M tools, each requiring a unique handshake.
The Model Context Protocol (MCP), introduced by Anthropic in late 2024, was designed to fix this. MCP is an open standard that lets any agent discover and call any tool through a single interface. It reached 97 million monthly SDK downloads by March 2026, with 5,800+ server implementations. When OpenAI committed to MCP support, it became genuinely cross-platform.
An MCP server exposes a system's capabilities in a way the agent can discover and interpret automatically. The agent asks "What tools are available?" and gets a structured answer it can reason about. The difference between handing someone a labeled toolbox and handing them a bag of unmarked metal objects.
N-times-M connector burden
Every new tool multiplied integration, auth, error handling, and parsing work across every agent.
MCP standardization
A shared discovery-and-tooling protocol reduces proprietary connector drag and improves interoperability.
Why it matters
Production teams need agents to discover capabilities through a contract, not through custom one-off glue every time.
Why Tool Calls Fail
Having tools isn't enough. In production, tool calls are the most common point of failure. The tool might be down. The API might return malformed data. The agent might call the wrong tool, pass the wrong parameters, or attempt a destructive action when it meant to query.
Reliable tool use requires:
- Input validation before calls
- Output parsing and verification (APIs return surprises more often than docs admit)
- Retry logic with exponential backoff for transient failures
- Permission boundaries restricting which tools are available in which contexts (principle of least privilege, applied to AI)
- Timeout handling so an agent waiting forever doesn't silently stop working
The fastest way to lose trust in a production agent is letting it perform an irreversible action without confirmation. I still prefer explicit confirmation flows for any write or delete operation, even when it adds friction.
Validation before execution
Bad inputs and surprising outputs need to be handled as routine cases, not edge cases.
Bounded retries
Retry loops without backoff, limits, and circuit breaking are one of the fastest ways to burn tokens and trust.
Least privilege
Agents should not be able to call every tool in every context, especially when destructive actions exist.
Guardrails
Guardrails are the defense layer that rejects harmful inputs and prevents the agent from doing things it shouldn't. The trick is calibration. Too weak and threats get through. Too aggressive and you reject legitimate requests, making the agent useless.
Production guardrails operate in three layers:
Input guardrails inspect what goes in. They catch prompt injection attacks (attempts to hijack the agent's behavior through crafted input), filter abusive content, and validate that requests fall within the agent's scope.
Process guardrails monitor what the agent does while working. They enforce policies like "never access customer financial data without an audit trail" or "limit external API calls per task to prevent runaway costs."
Output guardrails check what the agent says or does before results reach the user. They catch hallucinated facts, block PII exposure, and verify regulatory compliance.
The Speed-Safety Tradeoff
Every guardrail adds latency, and users hate waiting. The emerging approach is risk-based guardrailing: low-stakes interactions (browsing help docs, drafting an email) run with lightweight async checks that execute in the background while the agent streams its response. If a violation is detected after delivery, a correction is issued. High-stakes interactions (executing a financial transaction, modifying production infrastructure) trigger synchronous multi-layer verification. The agent pauses, checks clear, then proceeds.
High-risk actions need stricter checks, escalation, and review. Low-risk actions can stay lightweight so the agent remains useful for routine traffic. Overly aggressive guardrails can block routine interactions and make the agent unusable for common cases.
The most mature organizations encode guardrails as explicit, version-controlled policy code. When a regulator asks how you prevent unauthorized recommendations, you point to a versioned policy file and its test suite, not to a prompt that says "please be careful." The difference between a prompt instruction and a programmatic guardrail is the difference between asking someone to drive safely and installing anti-lock brakes.
Fallbacks: Surviving When Things Break
Every subsystem will fail at some point. The model will hallucinate. Memory retrieval will return irrelevant context. A tool will time out. A guardrail will misclassify a legitimate request. Production isn't about preventing failure. It's about surviving it.
Fallbacks operate on three escalating levels:
Level 1: Cached intelligence. When a component fails, fall back to cached results from recent similar operations. Vector database goes down? Use the most recently cached context. Quality is slightly lower, service continues.
Level 2: Heuristic routing. If caches are stale, fall back to rule-based heuristics. Instead of the LLM choosing which tool to invoke, a keyword-matching system makes a simpler routing decision. Less intelligent, still operational.
Level 3: Degraded service. When all sophisticated pathways fail, default to a safe minimal mode. Route to a single model with a conservative prompt. Tell the user some capabilities are temporarily unavailable and offer to connect them with a human.
Level 1: Cached intelligence
Recent context or answers keep the service moving when live dependencies fail temporarily.
Level 2: Heuristic routing
Simpler rules replace more sophisticated reasoning when dynamic selection becomes unreliable.
Level 3: Degraded service
The system falls back to the safest minimum viable behavior and clearly communicates the limitations.
Circuit Breakers
The circuit breaker pattern prevents a failing component from taking down the system. When a tool fails repeatedly, the breaker trips open and the agent stops calling it for a cooling-off period instead of hammering a dead endpoint. After cooldown, a single test request goes through. Success resumes normal traffic. Failure re-opens the breaker.
This is standard in microservices and increasingly mandatory for production agents, because agents are particularly prone to retry loops. A model that receives an error may keep attempting the same failing action unless something external stops it.
Human Escalation
For irreversible actions (deleting data, sending money, publishing content) the most reliable fallback is a person. The key is designing the handoff without destroying throughput. The best implementations are async: the agent queues the action, notifies the approver, and continues other tasks while waiting. The worst force synchronous holds, turning a twenty-second task into a twenty-minute one. Getting this right is its own design discipline, and the organizations that master it gain a genuine competitive advantage.
Orchestration
Memory, tools, guardrails, and fallbacks are separate subsystems. Without orchestration connecting them, they're a pile of capabilities, not a system.
Early agent frameworks used simple linear chains: input, model, tool, output. Production workflows are rarely linear. A real agent might branch based on intent, loop through approvals, run parallel sub-tasks, pause for human review, resume hours later, and handle failures at any step.
Graph-based orchestration frameworks (LangGraph being the most prominent) model these workflows as directed graphs with cycles. Each node is a processing step, each edge is a conditional transition. Graph state can be persisted and restored, so an agent can be interrupted mid-workflow and resume without losing its place.
As agents grow, single monolithic agents give way to teams of specialized agents with a supervisor coordinating them. The rule: no agent calls another directly. All communication flows through the orchestration layer, which preserves observability and lets you swap components. This is the same reason microservices communicate through well-defined APIs rather than reaching into each other's databases.
Observability is non-negotiable. Which model calls were made, which tools invoked, which guardrails fired, how long each step took, what the agent decided at each branch. When an agent makes a bad decision at 3 AM and a customer complains at 9 AM, the trace log is what lets you reconstruct exactly what happened and why.
What This Looks Like Under Pressure
A health insurance administrator we worked with deployed an AI agent for first-line member inquiries: benefit lookups, claim status, provider searches. About 40,000 inquiries per month. The pilot ran beautifully for eight weeks.
Then three things happened in one week. Their claims API hit intermittent 503 errors during a vendor migration. A new state regulation required specific disclosures on mental health benefit communications. And a member discovered they could get the agent to reveal other members' claim amounts through crafted questions.
The timeline on the API issue is worth spelling out. The 503s started Monday afternoon, about three per hour. By Tuesday morning they were hitting 30 per hour. Our initial assumption was that the vendor migration had a rollback plan and we'd just ride it out. That was wrong. The vendor's migration took eleven days, and the error rate fluctuated unpredictably the entire time. The circuit breaker on the claims API tripped automatically after three consecutive failures. The agent fell back to cached claim status with a disclosure that information might be up to 30 minutes stale. Level 1 degradation, no downtime.
The new regulation was addressed in four hours by adding a policy-as-code output guardrail. No model retraining, no prompt changes, just a new rule.
The prompt injection was patched by strengthening input guardrails. Episodic memory logs provided a full audit trail of exactly which data was exposed, enabling targeted breach notifications.
Total downtime across all three incidents: zero. The team estimated that without the fallback and guardrail architecture, the claims API failure alone would have caused 6-8 hours of complete agent unavailability, affecting roughly 1,200 member interactions.
Claims API instability
The agent degraded safely to cached claim status with disclosure instead of hard failing through an eleven-day vendor migration.
Regulatory change
A new output guardrail enforced required disclosure language in hours without retraining.
Prompt injection patch
Stronger input guardrails and episodic audit logs made the exploit traceable, containable, and reportable.
Common Misconceptions
"A bigger model fixes production problems." A more capable model can make production problems worse. Larger models hallucinate more confidently and use tools more creatively, including in ways you didn't intend. Production reliability comes from the architecture around the model.
"Guardrails are just prompt instructions." Writing "do not share personal information" in a system prompt is a suggestion. A code-level filter that redacts PII before the response reaches the user is a guardrail. The model might ignore a prompt instruction. It can't ignore code.
"Memory means stuffing conversation history into the prompt." Most common memory anti-pattern. Dumping the full conversation wastes tokens, drowns relevant info in noise, and eventually exceeds the context window. Production memory is selective retrieval weighted by relevance and recency.
Model capability is not architecture
A stronger model does not compensate for missing boundaries, missing memory discipline, or missing fallback behavior.
Policy must be enforceable
Prompt instructions are advisory. Code-level controls are what make safety operationally real.
Memory is retrieval, not dumping
Long transcript stuffing burns tokens and attention. Production memory is selective and relevance-weighted.
Takeaways
- Production agents need four subsystems: memory, tools, guardrails, and fallbacks. Skipping any one is why most agents die between pilot and production.
- The most common production failure is unhandled tool errors. Input validation, circuit breakers, and structured error handling eliminate the majority.
- Guardrails must be programmatic, version-controlled code, not prompt-level suggestions. Organize them as input, process, and output layers scaled to interaction stakes.
- Start with tool integration and input guardrails, add memory next, build fallback logic before you scale traffic.
- Human-in-the-loop is a feature, not a fallback. For irreversible actions, design an async handoff so throughput survives.
