The New AI Application Stack: What Comes After the LLM

For a while, "add AI to the product" meant adding one box: a call from the application straight to a model. That mental model was never wrong exactly, it was just describing a demo, not a system. A real AI application running in production today has grown a genuine architecture, eight distinct layers, each with its own failure modes, its own vendors, and its own reasons to page someone at 2 AM.

This is a full map of that stack, top to bottom, what each layer's actual job is, and where to go for the deep dive on any one of them. Think of this as the reference architecture the rest of this site's engineering coverage sits underneath.

The Stack

Application

The layer users actually see: the interface, the business logic wrapping around the AI functionality, the parts of the product that would exist in some form even if the AI component were swapped out entirely. One genuine inversion worth naming: in a lot of modern AI products, this is now the smallest layer by engineering effort, not the largest. The interesting work moved down the stack.

Agent / Workflow

The orchestration layer deciding what happens next: a fixed sequence of steps for predictable tasks, or a genuine agent loop, reason, act, observe, reason again, for tasks that need real flexibility. This is the layer that turns a single model call into a system capable of multi-step tasks, and it's covered in full architectural depth in AI Agents Are Not Chatbots, with the specific question of when this layer should fan out into several coordinating agents instead of one covered in Multi-Agent AI Systems, and a hands-on build of one concrete shape of that pattern, a Planner, a Worker, and an independent Reviewer, in Build a Multi-Agent System.

Model Router / LLMs

No single model is simultaneously the fastest, cheapest, most capable, and most specialized option for every request a real application makes. This layer's job is deciding, per request, which model actually fits, fast and cheap for simple requests, capable and slow for the ones that need it, specialized entirely for vision or code where a generalist underperforms. The full case for why this has become its own architectural layer, not just a model-selection afterthought, is in Why Companies Are Moving From Single-Model AI to Multi-Model AI Architectures.

RAG / Memory / Knowledge Graph

Three related but genuinely distinct components, often bundled together, that together determine what a model actually knows beyond whatever it learned during training.

Tools / MCP / APIs

How the system reaches past the model into the real world: databases, internal systems, external APIs, code execution. The standardized protocol increasingly used for this connection, and the architectural reasoning behind treating it as infrastructure rather than one-off integration code, is covered in MCP Explained.

Security / Guardrails / IAM

Three closely related concerns that belong in one layer because they answer one question together: what is this system, and whoever or whatever is currently acting through it, actually allowed to do. This spans identity and authorization for the agent itself, covered in AI Agent Identity, with the hands-on version of scoping that authority down to a real API, one tool per capability, tiered by blast radius, in How to Give AI Agents Access to APIs Without Giving Them Too Much Power; the full threat surface an agent introduces, covered in AI Agent Security and, for the specific mechanics of prompt injection, in Prompt Injection and Agent Hijacking; and the risk sitting even further upstream, in the models and dependencies the whole system was built on, covered in AI Supply Chain Security.

Evaluation / Observability

How anyone actually knows whether the seven layers above are working, before a user has to report that they aren't. Evaluation answers whether a specific change made the system better or worse, measured against a real baseline, covered in LLM Evaluation. Observability answers what actually happened inside any single request, which is what makes a specific wrong answer debuggable instead of a mystery, covered in AI Observability.

Inference Infrastructure

The compute foundation everything above actually runs on, and the layer where cost and latency get decided in practice, not in theory. GPU utilization, batching, caching, and the real economics of running models at scale are covered in The AI Inference Revolution.

Why the Mental Model Matters More Than the Diagram

Once "AI system" means eight layers instead of one box, a lot of decisions that used to feel like technology choices become architecture choices: not "which model," but "which layers does this specific product actually need, and in what order should we build them."

A Map, Not a Checklist

Not every product needs every layer built out fully on day one. A simple internal tool might genuinely be fine without a model router or a knowledge graph. But knowing which layers you're deliberately deferring is a very different position from not knowing they exist at all, and that difference is usually the one separating a system that scales from one that quietly breaks the first time it's actually tested.

The Bottom Line

The model was never the whole system. It was always the layer that got the headlines, sitting in the middle of a stack that, in any real production deployment, has seven other layers doing just as much work to keep it useful, safe, affordable, and debuggable. Treating "application plus LLM" as the architecture is how a team ends up building layer six for the first time during an incident, instead of before one.

The LLM was the beginning of this stack, not the whole of it. Everything else in this series is the rest of the diagram.