RAG without the hype: the mechanism decides the failure modes

A RAG system inherits the properties of the mechanism it retrieves with. If that mechanism compresses each text into a point and compares by cosine, there is a family of queries it will get wrong, and you can enumerate them before you have a single user.

That is not a hunch: it is what retrieval benchmarks have shown since BEIR (Thakur et al., 2021), where dense models lose to BM25 the moment they step outside the domain they were trained on. The mechanism takes a short while to explain and decides almost everything that follows.

How the machine works

An embedding model is a function. It takes a text of variable length and returns a fixed-length vector: between 384 and 3072 numbers, depending on the model. Four hundred tokens of documentation go in, 1,024 floats come out.

That is lossy compression. And the loss is not random: the model is trained to preserve one single property — that two texts with similar meaning end up close under cosine. Everything else gets thrown away. Syntax, order, negation, the difference between two codes that differ by one digit.

Then comes the index. An HNSW does not return the exact k neighbors: it returns k approximate neighbors, with a recall that you configure and that almost nobody measures.

The retriever is not searching for the answer to your question. It is searching for the texts whose numeric summary resembles the numeric summary of your question. Those are not the same thing, and all of RAG engineering lives in that gap.

What follows from the mechanism

Four consequences. Each one is a failure mode you can anticipate without running anything.

The chunk is the atomic unit of retrieval. You cannot retrieve half an idea. If the table header landed in chunk 7 and the row with the data landed in chunk 9, there is no top-k that brings back the complete answer. Chunking is not "split every 512 tokens": it is deciding what counts as a unit of meaning, and then repairing the context you destroyed by splitting. Anthropic measured the repair: prepending 50 to 100 tokens of generated context to each chunk dropped the top-20 retrieval failure rate from 5.7% to 3.7% — 35% less.

Semantic similarity is not exact match. E-4032 and E-4023 share tokens, share context and show up in nearly identical sentences. In embedding space they are stuck together. BM25 tells them apart without effort, because it counts terms instead of interpreting them. This is the concrete case behind the BEIR result cited above: where the exact term matters, counting terms wins.

The retriever's ranking is cheap and bad, by construction. A bi-encoder encodes the query and the document separately; they never see each other. A cross-encoder puts them together in the same forward pass, and there is cross-attention between the question and the text. It ranks vastly better and costs one forward pass per candidate. The whole architecture falls out of that: you retrieve 50 with the cheap thing, reorder 50 with the expensive thing, send 5 to the model.

Position in the prompt matters. Liu et al. measured a U-shaped curve in "Lost in the Middle" (arXiv:2307.03172, TACL 2024): with GPT-3.5-Turbo and 20 documents in context, accuracy drops on the order of 20 points when the relevant document sits in the middle instead of at the start. With 30 documents the drop is larger, and the exact magnitude depends on the model — the pattern repeats, the number does not. If you build the prompt sorted by descending score and put the best one first, you are playing along with the mechanism. If you sort by date, you are not.

The lab

Minimal corpus, in Spanish, with the traps built in. Install rank-bm25, sentence-transformers and numpy.

# retrieval.py
import re
import numpy as np
from rank_bm25 import BM25Okapi
from sentence_transformers import SentenceTransformer, CrossEncoder

DOCS = [
    "Devoluciones: el comercio puede solicitar la devolución total o parcial de un pago acreditado dentro de los 180 días corridos.",
    "Códigos de rechazo: E-4032 significa que la tarjeta no admite pagos en cuotas. E-4023 significa fondos insuficientes.",
    "Conciliación: el archivo se publica todos los días a las 06:00 ART e incluye los movimientos del día anterior.",
    "Webhooks: reintentamos hasta 5 veces con backoff exponencial. Un 2xx del endpoint cierra el reintento.",
    "Límites: el endpoint de creación de pagos acepta 100 requests por minuto por API key.",
    "Devoluciones parciales: se pueden encadenar hasta agotar el monto original del pago.",
]

def tok(s):
    return re.findall(r"\w+", s.lower())

bm25 = BM25Okapi([tok(d) for d in DOCS])
encoder = SentenceTransformer("intfloat/multilingual-e5-small")
M = encoder.encode([f"passage: {d}" for d in DOCS], normalize_embeddings=True)
reranker = CrossEncoder("BAAI/bge-reranker-v2-m3")  # multilingual on purpose

def dense(q, k=5):
    v = encoder.encode(f"query: {q}", normalize_embeddings=True)
    return np.argsort(-(M @ v))[:k].tolist()

def sparse(q, k=5):
    return np.argsort(-bm25.get_scores(tok(q)))[:k].tolist()

def rrf(*listas, k=60):
    # Reciprocal Rank Fusion, Cormack et al. SIGIR 2009. k=60 is the constant from the paper.
    acc = {}
    for lista in listas:
        for pos, doc in enumerate(lista, start=1):
            acc[doc] = acc.get(doc, 0.0) + 1.0 / (k + pos)
    return sorted(acc, key=acc.get, reverse=True)

def hibrido(q, k=5):
    return rrf(dense(q, 20), sparse(q, 20))[:k]

def con_rerank(q, k=3, pool=20):
    cands = rrf(dense(q, pool), sparse(q, pool))[:pool]
    scores = reranker.predict([(q, DOCS[i]) for i in cands])
    return [i for _, i in sorted(zip(scores, cands), key=lambda p: -p[0])][:k]

And the eval, which is the part almost nobody writes. It measures two things, not one: recall and milliseconds. Without the second number, "rerank aggressively" is a recommendation with no other side of the scale.

# eval.py
import statistics
import time
from retrieval import dense, sparse, hibrido, con_rerank

GOLD = [
    ("¿qué quiere decir el error E-4032?", {1}),
    ("¿hasta cuándo puedo devolver un pago?", {0, 5}),
    ("¿a qué hora sale la conciliación?", {2}),
    ("¿cuántos reintentos hace el webhook?", {3}),
    ("cuántos requests por minuto aguanta crear pagos", {4}),
]

def recall_at_k(fn, k):
    return sum(bool(gold & set(fn(q, k))) for q, gold in GOLD) / len(GOLD)

def ms_por_query(fn, k, repeticiones=5):
    fn(GOLD[0][0], k)  # warm-up: the first call loads weights and does not count
    tiempos = []
    for q, _ in GOLD:
        for _ in range(repeticiones):
            t0 = time.perf_counter()
            fn(q, k)
            tiempos.append((time.perf_counter() - t0) * 1000)
    return statistics.median(tiempos)

for nombre, fn in [("denso", dense), ("bm25", sparse),
                   ("híbrido", hibrido), ("híbrido+rerank", con_rerank)]:
    print(f"{nombre:16} recall@3={recall_at_k(fn, 3):.2f}  "
          f"mediana={ms_por_query(fn, 3):7.1f} ms")

Swap E-4032 for E-4023 in the query and watch what each retriever does. That thirty-second experiment is worth more than three days of prompt tuning.

Where it breaks

The English reranker. The cross-encoder that shows up in every tutorial is ms-marco-MiniLM-L-6-v2, trained on MS MARCO, which is an English corpus. On Spanish documentation it does not reorder anything for you: it dirties up the top-3 that the hybrid had already gotten right. This is a case where adding a stage "that improves everything" makes the system worse, and you will never see it if you do not measure. That is why I use bge-reranker-v2-m3 in the lab.

The index lies. This is the most expensive one of all and almost nobody has it on their list. You update a document, the pipeline reindexes and inserts the new chunks — and the old ones are still alive in the index because nobody deleted them. From that point on the system answers with total confidence using a policy that was repealed. No answer-quality eval catches it, because the answer is coherent, well written, cites its source and is wrong. The mitigation is boring and it works: store a doc_id and a hash of the source in every chunk, and delete by doc_id before inserting. A DELETE ... WHERE doc_id = ? ahead of every upsert buys more reliability than any reranker.

Negation and time. "What does the policy NOT cover?" retrieves the chunk that says what it does cover, because in embedding space they sit almost on top of each other. "What is the current version of the endpoint?" carries no semantic signal of recency at all: the model does not know which document is newer. You fix that with metadata and index filters, not with more embeddings.

High recall, low precision. Pulling 20 chunks so you do not miss the answer means feeding the model 18 irrelevant chunks. That is not free: it is exactly the "Lost in the Middle" scenario, and on top of that it hands the model material to hallucinate with while looking properly sourced.

The experiment that makes it obvious is reproducible in an afternoon: raise the chunk size and measure both ends. The retriever's recall@5 goes up — bigger chunks contain more answers, that is arithmetic — and answer accuracy goes down, because every chunk drags more irrelevant text along with the fragment that was useful. Retriever recall and system accuracy move in opposite directions, and if you measure only one you will optimize toward the wrong place with evidence backing you up. Measure both, always, in the same sweep.

The math

The public numbers, so the cost discussion stops being a matter of faith. List prices as of August 2026: Voyage charges USD 0.06 per million tokens on voyage-3.5 and USD 0.05 per million on rerank-2.5. Claude Opus 5 charges USD 5 per million input tokens. Re-check them before you quote them: inference prices move several times a year.

A corpus of 50,000 chunks of 400 tokens: 20 million tokens. Indexing the whole thing costs USD 1.20. Reindexing the whole thing costs another USD 1.20. Embeddings are free in practical terms — stop optimizing there.

Now the query path. Reranking 50 candidates of 400 tokens is 20,000 tokens per query: USD 0.001. At 10,000 queries per day, USD 10 a day. And the final prompt with 8 chunks is around 3,500 input tokens: USD 0.0175 per query, USD 175 a day. The generator costs seventeen times the reranker.

That gives you the decision that matters: it is cheaper to rerank aggressively and send 5 chunks than to skip the reranker and send 20. Reranking is not a quality luxury, it is a cost optimization. And add this: the retrieved chunks change on every request, so that part of the prompt never caches. Prompt caching saves you the system prompt, not the context.

The four axes of the decision

Before you build anything, the question is not which vector store you pick. There are four axes, and three of them can get you out of the problem entirely.

Corpus size and cardinality. Fewer than about 100 stable documents does not belong in an index: it belongs in the prompt. A model with a million-token context swallows your entire policy manual whole — no retriever, no chunking, no retrieval eval and no failure surface you do not control. RAG starts to pay when the corpus does not fit, or when stuffing all of it into every request costs more than searching it.

Volatility. A corpus that changes every six months and one that changes every hour are two different systems. The second one needs deletion by doc_id, incremental reindexing and an alarm for when indexing lag grows. If you are not going to build that, do not put RAG on top of a living corpus: you will serve stale answers that look current.

Error tolerance. A weird answer in an internal chat gets corrected with one message. A weird answer about a contractual condition has a cost that one message does not correct. That axis defines how much evidence you buy before launching, and it defines whether the system can answer on its own or has to show the document and let a person decide.

Vocabulary gap. This is the axis that decides whether you need the dense half at all. If your users write with the same words that are in the documents — technical terms, codes, proper nouns — BM25 alone is enough, and embeddings only add latency and a new source of errors. If they ask "can I get my money back?" and the document says "credit reversal", that is where the dense half wins something real. Measure it, do not assume it: it is exactly what the eval above compares.

When plain search is enough

RAG is justified when the question cannot be resolved by exact term. When it can, RAG adds latency, cost and a new source of error.

If your users know what they are looking for and search by term — a SKU, a case number, a customer name — a tsvector in Postgres with a GIN index gives them a better result, in 5 milliseconds, with no per-query cost and with filters they control. Putting an LLM in the middle there means adding latency, cost and a hallucination surface to solve something that was already solved.

RAG pays off when the answer is not in any single document but has to be synthesized out of three. That is the criterion. If you can point at a document and say "the answer is this one", go fetch that document and show it.

What I'd do

Eval first: 30 to 50 real questions with the correct chunk_id annotated by hand. Without that you do not have a system, you have a demo.

Then, in this order, and each step with its stop condition written down before you run it:

• BM25 alone, as the floor. If it gives you 0.90 recall@5 on your golden set, stop there. Do not add embeddings, do not stand up a vector store, do not add a dependency with 500 MB of weights to gain two points your generator is not going to use.
• Hybrid with RRF. It is ten lines of code. Stop condition: if it does not beat BM25 by more than the noise of your golden set — with 40 questions, two or three points are noise — stay with BM25 and save yourself half the latency.
• Reranking, in the right language. Stop condition: look at both columns of the eval together. If recall goes up but it adds 300 ms per query and your product is a synchronous chat, the decision is a product decision, not a retrieval one.
• Contextual chunking. Last, because it is the most expensive to operate: it forces an LLM pass over the entire corpus and a redo of that pass on every reindex. Stop condition: only if error analysis shows the failures come from context lost at split time, not from vocabulary or ranking.

Each step tells you whether the next one is worth it, and the mechanism tells you in advance what to expect from each one.