How RAG works: query, retrieve, and generate

Technical ExplainersSeptember 11, 20267 min read
Bright Jasper
Bright Jasper

Author

Ask a large language model a question about your company's internal onboarding process, and it will confidently make something up. Not because it's broken — because it was never trained on your onboarding process. This is the problem retrieval-augmented generation, or RAG, exists to solve.

RAG is an architecture that connects a language model to an external knowledge base at the moment a query comes in, so the model can ground its answer in something it wasn't trained on. It's become the default pattern for building AI systems that need to answer questions about internal documentation, product catalogs, or anything that changes faster than a model's training cycle.

Here's what's actually happening under the hood.

What problem does RAG solve?

RAG solves the frozen-knowledge problem: LLMs only know their training data. RAG retrieves relevant external documents at query time and feeds them as context, so the model answers from your data — not its memory.

A language model's knowledge is frozen at training time. It doesn't know about your product changes from last week, your internal wiki, or the incident report your team filed yesterday. Fine-tuning can adjust a model's tone or format, but it doesn't solve this — the model still doesn't know your data, and retraining every time something changes isn't practical.

RAG sidesteps this by not asking the model to know the answer. Instead, it retrieves the relevant information at query time and hands it to the model as context, alongside the question. The model's job shifts from "recall this from memory" to "read this and answer."

How does RAG work? (Two pipelines)

RAG uses an offline ingestion pipeline (chunk → embed → store in vector DB) and a real-time query pipeline (embed query → search vector DB → retrieve chunks → augment prompt → generate answer).

Most RAG systems are really two separate pipelines working together: an offline ingestion pipeline that runs ahead of time, and a real-time query pipeline that runs when a user asks something.

The ingestion pipeline takes your source documents, breaks them into smaller pieces (chunks), converts each chunk into a numerical representation (an embedding) using an embedding model, and stores those embeddings in a vector database. This happens once, or whenever your source data changes — not on every query.

The query pipeline is what happens when a user actually asks a question. The query itself gets converted into an embedding using the same method, the vector database is searched for chunks whose embeddings are mathematically closest to the query's embedding, and the most relevant chunks are pulled out and inserted into the prompt sent to the language model.

The model then generates its answer using both the original question and the retrieved chunks as context — which is the "augmented" part of retrieval-augmented generation.

Why does naive RAG fail? (Chunking issues)

Fixed-size chunking cuts sentences in half, giving the model incomplete context. Semantic, parent-child, or sliding-window chunking preserve meaning boundaries for better retrieval.

The single biggest source of retrieval errors in a basic RAG setup isn't the language model — it's how the source documents were chunked in the first place. Splitting documents into fixed-size blocks (say, every 500 tokens) is simple to implement, but it routinely cuts sentences and ideas in half, so the retriever ends up handing the model incomplete context.

A few more deliberate strategies exist:

  1. Semantic chunking — Split at natural boundaries (paragraphs, headers). Best for structured docs.
  2. Parent-child chunking — Small chunks for retrieval, expand to parent section for context. Best for dense manuals.
  3. Sliding window chunking — Overlap chunks 10–20%. Best for conversational data.

Which strategy fits depends on the source material: structured documentation tends to do well with semantic chunking, dense technical manuals often need parent-child chunking, and conversational data like support tickets works reasonably well with sliding windows.

What does an augmented prompt look like?

An augmented prompt injects retrieved chunks as context: Context: [chunks] Question: [query] Instruction: Answer using only the context above. The model then generates from that context.

Once relevant chunks are retrieved, they're not just appended randomly. A typical augmented prompt looks something like:

Context: [retrieved chunk 1] [retrieved chunk 2] [retrieved chunk 3]
Question: [original user question]
Instruction: Answer using only the context above.

The language model then generates a response grounded in that context rather than its training data alone. Some systems add a ranking step before this — scoring retrieved chunks by relevance and discarding the weaker ones — since handing the model too much irrelevant context can dilute the quality of its answer as easily as handing it too little.

Where does RAG still fail?

Failures are usually retrieval errors, not generation. Wrong chunks = confident wrong answers. RAG reduces hallucination only if retrieval quality is high; poor chunking or mismatched embeddings still hallucinate.

RAG reduces hallucination, but it doesn't eliminate it, and the failure mode is usually retrieval, not generation. If the retriever pulls the wrong chunks — because the embedding model didn't capture the query's actual intent, or the chunking split apart the information that mattered — the language model will still generate a confident, fluent answer. It'll just be confidently wrong about the wrong context, not the absence of context.

This is worth being precise about: RAG's reputation for "reducing hallucination" is really a claim about retrieval quality, not about the language model becoming more honest. A RAG system with poor chunking or a mismatched embedding model can still hallucinate freely — it's just hallucinating on top of bad context instead of no context.

When should you not use RAG?

Avoid RAG for small, stable knowledge bases that fit in a prompt. It adds latency, vector DB infrastructure, and retrieval failure surface. Use RAG when data is large, frequently changing, or exceeds context windows.

RAG is not a universal fix. It adds latency (a retrieval step before generation), infrastructure (a vector database, an embedding pipeline), and a new failure surface (retrieval quality) to a system that didn't have one before. If the knowledge you need the model to use is small, stable, and could reasonably fit in a prompt directly, you may not need RAG's ceremony to solve the problem — a well-designed prompt might be enough.

RAG earns its complexity when the knowledge base is large, changes frequently, or is too big to fit in a single prompt. For a single company wiki, a static FAQ, or documentation that rarely changes, that complexity may not pay for itself.

A note on the numbers: you'll find plenty of blog posts citing precise percentage improvements from adding RAG (accuracy gains, hallucination reduction rates). Most of these figures trace back to a single vendor's internal benchmark rather than an independently reproduced study, so I've left them out here rather than repeat an unverified number as settled fact. If you're evaluating RAG for a specific use case, the more reliable path is benchmarking retrieval quality against your own data rather than borrowing someone else's percentage.

References