Anatomy of an LLM application

The big idea

A weekend demo of an AI feature is one function: take the user's text, send it to a model, print the answer. A product that thousands of people use every day has a lot more around that call. It has to remember the conversation, look up your company's documents, check an order in a database, stream the answer to a phone, survive the model provider having a bad hour, and let you find out later why a customer got a wrong answer.

So the model is one component among many. Think of a restaurant. The chef is the star, but diners never talk to the chef. A waiter takes the order (the UI), the head of the kitchen decides what gets cooked and in what order (orchestration), the pantry supplies ingredients (retrieval), the ovens and grills do work the chef asks for (tools), the order book remembers every table (storage), and the manager's log records what went out and what came back (observability). A brilliant chef in a badly run kitchen still serves cold food.

This lesson names those parts, follows one request through them, and shows what each part is responsible for. The rest of the module zooms in on the parts that make or break a product: the interface, tracing, budgets and shipping changes safely.

How it works

UI: where expectations are set

The user interfaceThe screens, chat boxes, buttons and notifications through which a person uses the product. does more than show text. It streams tokens as they arrive so a 6-second answer feels quick, shows sources, offers retry and undo, and asks for confirmation before anything with consequences. These patterns are the subject of UX patterns for AI . Streaming usually rides on server-sent events or WebSockets ( Real-time: polling, long polling, SSE and WebSockets ).

Orchestration: the part you actually write

The orchestration layerThe application code that turns a user request into model calls: it gathers context, builds the prompt, runs the tool loop, validates outputs and decides what to return. is your backend code for the AI feature, and it holds most of the product logic. It decides which context to fetch, assembles the prompt from instructions, history and retrieved passages, calls the model, runs the tool loop from The agent loop (ReAct) , validates the output against a schema, and handles errors. It can be a few hundred lines of plain code or built on a framework. Either way, keep prompts as versioned files, not strings scattered through the code, because you will change them often.

Model API, ideally behind a gateway

The model itself is usually a call over HTTP to a provider, or to a model you host. Production apps often put a thin model gatewayA shared service in front of model providers that handles API keys, retries, fallbacks to other models, rate limits, caching and usage accounting. in front of it: one place for keys, timeouts and retries ( Timeouts, retries, backoff and jitter ), fallback to a second model when the first is down, rate limits, prompt caching ( Prompt caching and cost control ) and per-team usage tracking. It is the same idea as an API gateway in classic systems ( API gateway and backend-for-frontend ).

Retrieval: your data, at question time

RetrievalFinding the passages from your own documents or databases that are relevant to the current question, so they can be added to the prompt. gives the model facts it was never trained on: your help center, contracts, tickets. It is the RAG pipeline from Why RAG? , with an ingestion job that keeps the index fresh as documents change, and permission filters so a user can only retrieve what they are allowed to read.

Tools: actions with real effects

Tools are ordinary functions the model can ask for, such as looking up an order, issuing a refund or running code. The model only proposes; your code checks the arguments and permissions, runs the call and returns the result. Anything that changes the world needs limits and, for risky actions, a human approval step ( Coding agents and computer-use agents ).

Storage: conversations and state

LLM APIs are stateless: each call sees only what you send. So your app stores conversations, user settings, long-term memories ( Memory for agents ), uploaded files and the state of long agent runs. Keeping state in storage, not in the server's memory, also lets you run many identical servers behind a load balancer ( Stateless vs stateful services ).

Observability: what actually happened

ObservabilityRecording what a system did, for every request, in enough detail to explain its behavior afterwards: logs, metrics and traces. for an LLM app means recording the exact prompt, retrieved passages, tool calls, output, token counts, latency and cost of every step. Without it, a user report like "the bot told me the wrong refund window" cannot be debugged, and you cannot build evals from real failures ( Why evals are the core skill ). LLM observability and tracing covers tracing in depth.

Try it

1. Which job did you hesitate on? Most confusion is between orchestration and the gateway: the gateway is shared plumbing for every feature, orchestration is the logic of one feature.
2. Now read the same layers as code. Every part is a few lines here, with a fake model so it runs anywhere:

# Every layer of an LLM app in one file, with a fake model so it runs offline.
import json, re

DOCS = {  # retrieval: a tiny knowledge base
    "returns": "You can return items within 30 days for a full refund.",
    "shipping": "Each order ships in 3 to 5 business days.",
}
ORDERS = {"A17": "shipped", "B42": "delivered"}  # data behind a tool
STORE = {}   # storage: conversation history per session
TRACE = []   # observability: one record per step

def log(step, **info):
    TRACE.append({"step": step, **info})

def retrieve(q):
    words = set(re.findall(r"\w+", q.lower()))
    best = max(DOCS, key=lambda d: len(words & set(re.findall(r"\w+", DOCS[d].lower()))))
    log("retrieve", query=q, hit=best)
    return DOCS[best]

def order_status(order_id):  # a tool: ordinary code that the model can ask for
    log("tool", tool="order_status", order_id=order_id)
    return ORDERS.get(order_id, "not found")

def fake_model(prompt):  # stand-in for the model API
    log("model", prompt_tokens=len(prompt) // 4)  # rough: 4 characters per token
    parts = dict(line.split(": ", 1) for line in prompt.splitlines())
    ids = re.findall(r"\b[A-Z]\d+\b", parts["USER"])
    if "TOOL RESULT" in parts:
        return {"text": f"Your order is {parts['TOOL RESULT']}."}
    if ids:
        return {"tool": "order_status", "args": {"order_id": ids[-1]}}
    return {"text": parts["CONTEXT"]}

def handle(session, msg):  # orchestration: build prompt, call model, run tools
    history = STORE.setdefault(session, [])
    prompt = f"SYSTEM: be brief\nHISTORY: {history[-2:]}\nCONTEXT: {retrieve(msg)}\nUSER: {msg}"
    reply = fake_model(prompt)
    if "tool" in reply:
        prompt += f"\nTOOL RESULT: {order_status(**reply['args'])}"
        reply = fake_model(prompt)
    history += [msg, reply["text"]]
    log("store", session=session, messages=len(history))
    return reply["text"]

print(handle("s1", "Where is my order B42?"))  # the UI would show these
print(handle("s1", "Can I return it for a refund?"))
for record in TRACE:
    print(json.dumps(record))

Your order is delivered.
You can return items within 30 days for a full refund.
{"step": "retrieve", "query": "Where is my order B42?", "hit": "shipping"}
{"step": "model", "prompt_tokens": 27}
{"step": "tool", "tool": "order_status", "order_id": "B42"}
{"step": "model", "prompt_tokens": 32}
{"step": "store", "session": "s1", "messages": 2}
{"step": "retrieve", "query": "Can I return it for a refund?", "hit": "returns"}
{"step": "model", "prompt_tokens": 45}
{"step": "store", "session": "s1", "messages": 4}

3. Look at the trace. The first question needed a tool and two model calls; the second needed one. Notice that the second prompt is longer (45 rough tokens versus 27) because it carries the history from storage. That growth is why long conversations get slower and more expensive.

Going deeper

Where the complexity actually lives

Teams new to AI products expect the model call to be the hard part. In practice it is one line. The work is in the layers around it: getting the right context into the prompt, handling the model's mistakes, keeping the index fresh, and seeing what happened. A useful rule: if a behavior must always hold, put it in code, not in the prompt. "Never refund more than 200 dollars" belongs in the refund tool, where a confused or manipulated model cannot talk its way past it.

One request, many model calls

A single user message often triggers several model calls: a cheap one to classify the request or rewrite the search query, the main answer call, perhaps a tool round trip, and a check on the output. Each adds latency and cost, which is why Latency and cost budgets breaks a request into its parts and budgets each one.

Frameworks or plain code

Frameworks give you retrieval, tool loops and tracing out of the box, which is fast for a prototype. They also add layers between you and the prompt that is actually sent. Many teams start with a framework, then move the core loop to plain code once they need to see and control every token. Whatever you choose, make sure you can print the exact final prompt.

Trade-offs

Common mistakes

• Treating the model as the whole product, then discovering memory, permissions and logging after launch.
• Building prompts from strings spread across the codebase, so nobody knows which prompt is live.
• Putting hard business rules only in the prompt instead of enforcing them in tools and validation code.
• Keeping conversation state in one server's memory, so a restart or a second server loses it.
• Shipping without per-request traces, so a bad answer cannot be explained or turned into an eval case.

In the wild

Check yourself

Recap