LLMs: the right mental model

"Let's drop an LLM in here" gets discussed as a decision about AI. It's an architecture decision, and a far more specific one than it sounds: you're swapping a deterministic function, with fixed cost and bounded latency, for a probabilistic one, with variable cost, variable latency, and whose failure mode is returning a plausible, wrong answer without flagging it.

Everything else is implementation. This article is the mechanism that sentence follows from, and the four axes that come out of it.

What you're swapping out

Each of those four properties has consequences that come due at different moments, and that lag is what makes the decision feel free when you take it. Variable cost shows up in the third month's bill, once volume has grown. Variable latency shows up when someone measures p99 instead of the median. And the failure mode shows up in production, because a plausible, wrong answer doesn't fire an alert: it passes the shape tests, it looks good in the demo, and the only person who catches it is someone who already knows the right answer.

The point isn't that an LLM is a bad idea. It's that those four properties are the component's spec, not defects to be corrected, and you should decide with them in plain sight.

What you need isn't knowing how to train a transformer — it's knowing what guarantees it gives you and which ones it doesn't. That's where the four axes come from.

The mechanism

An LLM is an autoregressive next-token predictor over a context window. Three steps, in a loop.

One: tokenization. Your text is split against a fixed BPE vocabulary — decided before training, immutable afterwards — and each piece becomes an integer. The model never sees characters. It sees indices.

Two: forward pass. Those integers become vectors and pass through a stack of identical blocks. In each block, self-attention makes every position look at every position before it: quadratic cost in sequence length. That isn't memory. It's a dot product across all the positions, recomputed from scratch on every request.

Three: softmax and sampler. The last layer returns one number for every token in the vocabulary. Softmax turns them into a probability distribution. A sampler picks one. That token gets appended to the sequence and you go back to step one.

There's no database. There's no lookup. There's a giant matrix multiplication and a loaded die.

Three consequences follow, and they govern the decision.

It has no memory. The API is stateless. Every turn of a conversation resends the full history. If your chat has 20 turns, turn 20 pays for everything before it all over again. The input cost of a conversation grows quadratically with the number of turns, not linearly. Nobody models it that way when they estimate the budget.

It operates on tokens, not on facts or characters. A token is neither a word nor a letter. That's why the model fails at counting the letters in a word and is excellent at summarizing a contract: those aren't the same kind of task for it, even though both look equally easy to you. And if you want to know how many tokens your prompt takes up, count them with count_tokens against the model you're actually going to use — the vocabulary changes between model families, so a tokenizer from another provider will give you a number that isn't the one you get billed for.

It has no "I don't know" primitive. There is always a most-likely token. When it doesn't have the information, it doesn't return an error — it returns the most plausible continuation. What we call hallucination isn't a bug: it's the correct behavior of the mechanism applied to an input for which there was no answer in the distribution.

The four axes

Four axes, and the order matters: the first two are disqualifiers.

• Is the output verifiable by code? If you can write an assert over the answer, the LLM is a reasonable candidate. But look closely at what that assert verifies: a schema constrains shape, not existence. A made-up CUIT — the Argentine tax ID — with a correctly computed check digit passes any format validation and isn't in the registry. A SKU with the right prefix parses perfectly and doesn't exist in your catalog. Verification that's worth anything runs against a system that knows the truth — your database, the registry, the provider's API — not against a regular expression. If verification requires a human to read and form an opinion, you're buying a scaling problem.
• What's the blast radius of a plausible, wrong answer? Not of an obviously wrong one. Of one that looks right. Classifying support tickets: low. Deciding whether to approve a transfer: high. Drafting an email that sends itself: high and silent, which is the worst kind.
• What percentage of the input is repeated prefix? This axis decides the entire economics, and I unpack it below.
• Does the latency fit the path's budget? An LLM in the synchronous path of a checkout is a different architecture decision than an LLM in a nightly job. If the case tolerates minutes instead of milliseconds, the Batch API charges you half price.

If the first two come out badly, the other two don't matter. Stop there.

The numbers

Public Anthropic API prices, per million tokens:

• Claude Opus 5: 5 dollars input, 25 output. 1M-token window, up to 128K of output.
• Claude Haiku 4.5: 1 dollar input, 5 output. 200K window.

Now the part that changes the order of magnitude. Prompt caching: a cache read costs around 10% of the base input price, and a write costs 1.25× with a 5-minute TTL (2× if you choose the one-hour TTL). The break-even point with a 5-minute TTL is two requests.

Take an assistant over internal documentation: 40,000 tokens of stable prefix (system prompt plus operations manual), 200 tokens of question, 800 of answer, 50,000 requests a month, on Opus 5.

Without caching: 2,010 million input tokens at 5 dollars per million is 10,050 dollars, plus 1,000 of output. About 11,050 dollars a month.

With caching and a 95% hit rate: 625 dollars of writes, 950 of reads, 50 for the variable suffix, 1,000 of output. About 2,625 dollars a month.

Same application, same model, same answer quality. A difference of 8,400 dollars a month that depends entirely on where you put one field in the prompt.

What breaks

Caching is prefix match. Any byte that changes in the prefix invalidates everything after it. And this is the mistake that gets made every single time:

# Wrong. The timestamp is inside the cached prefix.
system = [{
    "type": "text",
    "text": f"Hoy es {datetime.now().isoformat()}\n\n{MANUAL_OPERATIVO}",
    "cache_control": {"type": "ephemeral"},
}]

That datetime.now() makes the prefix different on every request. The cache never gets read. No error, no warning, no log. You simply pay full price forever and the bill tells you 30 days later.

# Right. The prefix is byte-identical across requests; the volatile part goes at the end.
system = [{
    "type": "text",
    "text": MANUAL_OPERATIVO,           # 40,000 tokens, frozen
    "cache_control": {"type": "ephemeral"},
}]
messages = [{"role": "user", "content": [
    {"type": "text", "text": f"Fecha: {datetime.now().isoformat()}"},
    {"type": "text", "text": pregunta},
]}]

The check is one line, and it belongs on your dashboard, not in your head:

r = client.messages.create(...)
print(r.usage.cache_read_input_tokens)   # if this is 0 request after request, you're not caching
print(r.usage.cache_creation_input_tokens)
print(r.usage.input_tokens)              # careful: this is ONLY the uncached remainder

That last field misleads you, and it misleads you in the comfortable direction. input_tokens isn't the size of the prompt: it's the leftover that fell outside the cache. The total prompt is the sum of the three fields. If you look only at input_tokens to estimate consumption, you'll believe you're processing a fraction of what you actually process.

More things break for the same reason. Changing the tool set invalidates everything, because tools get rendered at position zero of the prompt — still true by default, though on Opus 5 there's now a beta path (mid-conversation-tool-changes-2026-07-01) that adds and removes tools mid-conversation without throwing away the cached prefix. Switching models mid-conversation invalidates everything, because caches are per model. And the minimum cacheable prefix isn't uniform: 512 tokens on Opus 5, 1024 on Sonnet 5 and Opus 4.8, 2048 on Opus 4.7, 4096 on Opus 4.6 and Haiku 4.5. The scale isn't monotonic across generations. A 3,000-token prompt caches on one model and silently doesn't cache on another.

Determinism doesn't exist either. This has exactly the same shape as the caching failures, which is why it goes here. temperature=0 doesn't mean "the model is deterministic": it means greedy decoding, picking the argmax of the logits. And the logits depend on how your request got grouped with others in the provider's batch, because floating-point reductions aren't associative — adding the same numbers in a different order gives results that differ in the last bit. With two candidates nearly tied, that last bit flips the argmax and the answer changes. Same prompt, same temperature, different output. On Anthropic's current models the point is even more final: temperature, top_p and top_k were removed as of Opus 4.7, and sending them returns a 400. The lever is gone. If you wrote a regression test that compares output byte for byte, you don't have a test: you have a flaky-build generator that will eat days of team time before anyone works out why.

The other predictable failure mode is positional. In Lost in the Middle (Liu et al., TACL 2024) the authors measured information retrieval as a function of where the data sat inside the context: accuracy is highest when the data is at the beginning or the end, and drops sharply when it's in the middle — even in models explicitly designed for long context. If your retrieval stuffs in 30 documents and the relevant one lands fifteenth, the failure is structural, not a prompt problem.

And there's one more that rearranges the mental model on the spot. In The Reversal Curse (Berglund et al., 2023) the authors showed that a model trained on "A is B" doesn't learn "B is A". The paper's example is the one that sticks with you: the model correctly answers who Tom Cruise's mother is and fails when you ask whose son Mary Lee Pfeiffer is. Same fact, reversed direction, and the asymmetry is enormous. There's no knowledge graph in there. There's a continuation function trained on an ordering of words, and that ordering matters.

When not to do it

Four cases where the answer is no, and I'll defend them:

• When a rule already exists. If the requirement can be written as an if, write it as an if. An LLM classifying by a criterion you've already specified is more expensive, slower and less reliable than the if. It sounds obvious, and yet it's the item on this list that takes longest to detect: an if that was wrongly replaced doesn't fail, it just costs money and adds delay.
• When the output isn't verifiable and the error is expensive. Without automatic verification, your quality system is hope. Hope doesn't scale to 50,000 requests a month.
• When volume is high and value per request is low. Running an LLM over every event of a high-volume stream is a decision you pay for in real dollars. Do the math beforehand, not afterwards.
• When you're using it as the orchestrator of a deterministic plan. If you look at your agent's trace and it's the same loop N times with inputs that fully determine the outputs, that's code with extra steps and a bill attached. Keep a single model call where the work is genuinely ambiguous, and hand the rest back to functions.

The right mental model doesn't tell you what you can build. It tells you what's going to fail, and that's worth more.

An LLM is a component with strange guarantees: probabilistic, stateless, expensive on repeated prefixes, blind to the middle of its own context, asymmetric about the facts it was taught, and incapable of saying it doesn't know. None of those properties is a defect to be corrected. They're the spec.

A system designed against that spec behaves the way you expect. One designed against the imagined spec behaves exactly as it should too — the surprise just arrives in production, with real money.