Retrieval augmented generation (RAG): how it works, with a working example

Retrieval augmented generation (RAG) is a pattern where your app first searches your own documents for passages relevant to a question, then gives those passages to a language model so it answers from them. This guide explains the moving parts, builds a minimal RAG pipeline in about 40 lines of Python, and covers chunking, evaluation and when to skip RAG.

What is retrieval augmented generation?

Retrieval augmented generation (RAG) is a way to make a language model answer questions from your own data. Before the model writes anything, your code searches a document collection, picks the most relevant passages, and pastes them into the prompt with an instruction like "answer using only this context."

The name comes from a 2020 paper by Patrick Lewis and colleagues, which combined a neural retriever over Wikipedia with a text generator and showed the result produced more specific and factual answers than the generator alone. Today "RAG" covers any system that retrieves first and generates second, whether the retriever is a vector search, a keyword index, a SQL query or a web search.

You use RAG when the model needs facts it was never trained on: your company wiki, product docs, support tickets, contracts, or anything newer than the model's training data. Current Claude models, for example, have a reliable knowledge cutoff of June 2026, per Anthropic's models overview. Anything after that, or anything private, has to come from you.

How does a RAG pipeline work?

A RAG pipeline has an offline indexing step and an online question step. Indexing prepares your documents once; the question step runs every time a user asks something.

Indexing (offline):

1. Load your documents (Markdown, PDFs, HTML, tickets).
2. Split them into chunks of a few hundred tokens each.
3. Turn each chunk into an embedding, a list of numbers that captures its meaning.
4. Store the chunks and their embeddings in a searchable index.

Answering (online):

1. Embed the user's question with the same embedding model.
2. Find the chunks whose embeddings are closest to the question's.
3. Put the top chunks into a prompt with the question.
4. Ask the LLM to answer from that context and cite which chunk it used.

If embeddings are new to you, the plain-English guide on what embeddings are covers similarity search in more depth.

A minimal RAG example in Python

The smallest useful RAG system fits in one file: embed a handful of documents, embed the question, take the best matches by dot product, and send them to Claude. This example uses Voyage AI for embeddings because Anthropic does not offer its own embedding model and points to Voyage in its docs. Any embedding provider works the same way.

pip install -U voyageai anthropic numpyexport VOYAGE_API_KEY="your-voyage-key"export ANTHROPIC_API_KEY="your-anthropic-key"

import numpy as npimport voyageaiimport anthropicdocuments = [    "Refunds: customers can request a full refund within 30 days of purchase.",    "Shipping: orders ship within 2 business days from our Rotterdam warehouse.",    "Warranty: hardware is covered for 24 months against manufacturing defects.",    "Support hours: Monday to Friday, 9:00 to 17:00 CET, by email only.",    "Returns: items must be unused and in original packaging to be returned.",]vo = voyageai.Client()          # reads VOYAGE_API_KEYclaude = anthropic.Anthropic()  # reads ANTHROPIC_API_KEYdoc_vectors = np.array(  # index step: embed every document once    vo.embed(documents, model="voyage-4", input_type="document").embeddings)def retrieve(question: str, k: int = 2) -> list[str]:    q = vo.embed([question], model="voyage-4", input_type="query").embeddings[0]    scores = doc_vectors @ np.array(q)  # Voyage vectors are normalized    top = np.argsort(scores)[::-1][:k]    return [documents[i] for i in top]def answer(question: str) -> str:    chunks = retrieve(question)    context = "\n".join(f"[{i + 1}] {c}" for i, c in enumerate(chunks))    response = claude.messages.create(        model="claude-sonnet-5-5",        max_tokens=500,        system=(            "Answer only from the numbered context. Cite chunk numbers like [1]. "            "If the context does not contain the answer, say you don't know."        ),        messages=[{            "role": "user",            "content": f"Context:\n{context}\n\nQuestion: {question}",        }],    )    return response.content[0].textprint(answer("Can I get my money back after three weeks?"))

Three details in this code matter more than they look:

• input_type is set to "document" when indexing and "query" when searching. Voyage's docs say not to omit it for retrieval, because it changes how the text is represented.
• Dot product equals cosine similarity here because Voyage returns vectors normalized to length 1. With other providers, check whether you need to normalize first.
• The system prompt gives the model a way out. Without "say you don't know," models tend to answer from general knowledge when retrieval misses, which is the main way RAG systems hallucinate.

For anything beyond a demo, move the vectors out of memory into a real index. Postgres with pgvector is the shortest path:

CREATE EXTENSION vector;CREATE TABLE chunks (  id bigserial PRIMARY KEY,  source text,  content text,  embedding vector(1024));CREATE INDEX ON chunks USING hnsw (embedding vector_cosine_ops);-- top 5 chunks by cosine distanceSELECT source, contentFROM chunksORDER BY embedding <=> $1LIMIT 5;

The <=> operator is cosine distance, and the HNSW index keeps the query fast as the table grows. The vector database guide covers when Postgres stops being enough.

How should you chunk documents?

Chunk along the structure of the document, not at fixed character counts. Split on headings first, then paragraphs, and keep each chunk a self-contained passage of a few hundred tokens with a small overlap.

Bad chunking is the most common silent RAG failure. A chunk that starts mid-sentence or separates a table from its header retrieves poorly and confuses the model even when it is retrieved. Rules that hold up in practice:

• Keep the title path. Prepend "Docs > Billing > Refunds" to each chunk so it carries its context.
• Never split tables, code blocks or lists across chunks if you can avoid it.
• Store metadata (source URL, date, product, access group) next to each chunk for filtering and citations.
• Deduplicate. Ten near-identical copies of the same paragraph crowd out everything else in the top results.

Anthropic's Contextual Retrieval technique goes one step further: before embedding, it asks Claude to write 50 to 100 tokens explaining where each chunk sits in its document, and prepends that to the chunk. In Anthropic's tests, contextual embeddings cut the top-20 retrieval failure rate by 35%, adding contextual BM25 keyword search cut it by 49%, and adding a reranker on top cut it by 67%.

How do you make retrieval more accurate?

Combine semantic search with keyword search, retrieve generously, then rerank. Embeddings are good at meaning but weak on exact terms like error codes, SKUs and names; BM25 keyword search is the reverse.

A retrieval stack that works for most teams:

1. Hybrid search. Run vector search and BM25 in parallel and merge the results (reciprocal rank fusion is a common, simple merge).
2. Retrieve wide. Pull about 100 to 150 candidates. Anthropic's reranking pipeline retrieves around 150 and keeps the top 20.
3. Rerank. A reranker model scores each candidate against the question and is far more precise than raw vector distance.
4. Pass enough context. Anthropic tested 5, 10 and 20 chunks and found 20 performed best in its setup. Modern models handle that much context well.
5. Filter by metadata before or during search, so a user never retrieves documents they are not allowed to read.

Retrieved text is also an injection channel. A document that says "ignore your instructions" ends up in the prompt like any other chunk, so treat retrieved content as untrusted data. The post on prompt injection explains the patterns.

How do you evaluate a RAG system?

Evaluate retrieval and generation separately, starting with retrieval. If the right chunk is not in the context, no prompt will fix the answer.

Build a small test set first: 30 to 100 real questions, each labeled with the chunk or document that contains the answer. Then measure:

Frameworks like RAGAS automate some of these with an LLM as judge. Use them, but read a sample of failures by hand every time you change chunking, embeddings or prompts. LLM judges have their own blind spots, and twenty minutes of reading failures usually tells you more than a dashboard.

Track the retrieval failure rate the same way Anthropic does in its write-up: the share of questions where the needed chunk is missing from the top results. It is the single number that tells you whether to work on retrieval or on generation.

When should you not use RAG?

Skip RAG when your knowledge fits in the prompt. Anthropic's guidance is that a knowledge base under about 200,000 tokens (roughly 500 pages) can go into the prompt whole, and prompt caching makes repeat requests faster and up to 90% cheaper, per its Contextual Retrieval post. With current Claude models accepting 1 million tokens of context, the "fits in the prompt" threshold is higher than it used to be, though cost per request grows with context size.

RAG is also the wrong tool when:

• You need a behavior, not facts. A consistent output format or tone is a prompting or fine-tuning problem. See fine-tuning vs RAG for the decision.
• The data is structured. Questions like "revenue by region last quarter" belong in SQL, not in vector search over CSV rows. Let the model write the query instead.
• An agent can search directly. Coding agents like Claude Code find what they need in a repository with grep and file reads, an approach often called agentic search. For codebases, that often beats a pre-built embedding index that goes stale with every commit.

The broader idea, deciding what information goes into the model's context and when, is covered in context engineering.

How to start a RAG project this week

Start small, measure early, and add complexity only where the measurements tell you to.

1. Check the size. Count tokens in your corpus. Under 200,000, try the whole corpus in a cached prompt first.
2. Write 30 test questions with known answers before writing any code.
3. Build the minimal version above with your real documents and structure-aware chunks.
4. Measure recall@20. If the right chunk is missing more than occasionally, fix chunking, add BM25, then add a reranker, in that order.
5. Tighten the prompt: answer from context, cite chunks, admit when the answer is missing.
6. Move to pgvector with metadata filters once you have more than a demo's worth of data.
7. Re-run the test set after every change and keep the scores in a spreadsheet.

If you want a coding agent to build and iterate on the pipeline while you are away from your desk, Orkio gives Claude Code an always-on cloud box that keeps working after you close your laptop.

Frequently asked questions