Fine-tuning vs RAG: how to choose, with examples and real costs

Use RAG when the model needs knowledge it doesn't have (your documents, fresh data); use fine-tuning when it needs a behavior that prompting can't reliably produce (a strict format, a narrow task on a small, cheap model). Most teams should try good prompting first. This guide compares all three with examples, costs and a decision checklist.

Fine-tuning vs RAG: what is the difference?

Fine-tuning vs RAG comes down to one question: does the model lack knowledge or behavior? Retrieval-augmented generation (RAG) adds knowledge at question time by searching your documents and putting the relevant passages in the prompt. Fine-tuning changes the model's weights by training it on examples, which shapes how it responds.

A simple way to remember it:

• RAG is an open-book exam. The model looks up the answer in material you hand it each time.
• Fine-tuning is practice. The model rehearses many examples until a pattern becomes habit.
• Prompting is instructions. You tell the model what to do, show it a few examples and give it the context it needs.

The two are not mutually exclusive, but the order of trying them matters. Start with prompting, add retrieval when knowledge is the gap, and fine-tune last.

Why should you try prompting first?

Try prompting first because it is the cheapest, fastest option and is often enough with current models. OpenAI's model optimization guide says "the prompt engineering process may be all you need in order to get great results for your use case," and recommends writing evaluations before you write prompts.

Two developments make prompting go further than it used to:

1. Long context windows. Anthropic's Contextual Retrieval article notes that if your knowledge base is "smaller than 200,000 tokens" (about 500 pages), you can often include all of it in the prompt and skip retrieval entirely.
2. Prompt caching. If the same large prefix (instructions, documents, examples) is sent on every request, you can cache it. In Claude's prompt caching docs, cache reads cost 0.1 times the base input price on most models (lower on some current models), with a 5-minute default lifetime that refreshes each time the cache is used.

A strong prompt for a knowledge task often looks like this:

You are a support assistant for Acme's billing product.Answer only from the documents below. If the answer isn't there, say"I don't know" and suggest contacting support@acme.com.Cite the document title for every claim.<documents>  <document title="Refund policy">...</document>  <document title="Plan limits">...</document></documents>Question: {user_question}

If that prompt, with the whole document set inside it, answers your evaluation questions well and the cost is acceptable, stop. You are done. See our prompt engineering guide and context engineering for techniques that get more out of this stage.

When should you use RAG?

Use RAG when the knowledge the model needs is too large for the prompt, changes often, must be cited, or has access rules. Retrieval puts only the relevant pieces in front of the model for each question.

Good RAG use cases:

• Support assistants over thousands of help articles and past tickets.
• Internal search across wikis, policies and contracts, with per-user permissions.
• Fresh data: product catalogs, prices, inventory or news that change daily.
• Answers with citations, where users or auditors need to check the source.
• Code assistants that search a large repository instead of reading every file.

How a RAG pipeline works

1. Chunk documents into passages of a few hundred tokens.
2. Embed each chunk into a vector and store it (see what embeddings are).
3. Retrieve the chunks most similar to the question, ideally combining vector search with keyword search.
4. Rerank the candidates and keep the best few.
5. Generate an answer from the question plus the retrieved chunks.

Retrieval quality decides answer quality. Anthropic's Contextual Retrieval research reports that combining embeddings with BM25 keyword search beats embeddings alone, because keyword search catches exact terms like error codes. Adding a short context summary to each chunk before embedding reduced failed retrievals by 49%, and adding reranking brought the reduction to 67% in their tests.

Where RAG falls short

• It can't change behavior. Retrieved documents don't teach the model a new output format or tone.
• Bad retrieval means bad answers. If the right chunk isn't found, the model answers from memory or says it doesn't know.
• It adds infrastructure. An index, an embedding pipeline, re-indexing jobs and monitoring.
• It adds tokens. Every retrieved chunk is input you pay for on each request.

Our guide to retrieval-augmented generation walks through a minimal working example.

When should you use fine-tuning?

Use fine-tuning when you need a behavior that careful prompting can't produce reliably, or when you want a smaller, cheaper model to match a larger one on a narrow task. OpenAI's guide lists the benefits over prompting alone: training on more examples than fit in a context window, shorter prompts, keeping proprietary examples out of each request, and training a smaller model for a specific task.

Good fine-tuning use cases:

• Strict output formats that prompting gets right most of the time, but not always, at high volume.
• Narrow classification or extraction: routing tickets, tagging transactions, extracting fields from a known document type.
• Distillation: teaching a small model to do one job that a large model does well, to cut latency and cost.
• Consistent style or tone that is hard to describe but easy to show in many examples.

Fine-tuning methods

For open-weight models, the Hugging Face PEFT library fine-tunes "a small number of (extra) model parameters," which significantly reduces compute and storage costs compared with full fine-tuning.

Why fine-tuning is a poor way to add knowledge

Fine-tuning on your documents rarely produces a model that answers reliably from them. The facts get blended into the model's general knowledge, it can't tell you where an answer came from, and every time a document changes you have to retrain. Use retrieval for facts.

What do prompting, RAG and fine-tuning cost?

Prompting costs only tokens, RAG adds search infrastructure and retrieved tokens, and fine-tuning adds data work, training runs and often higher per-token or hosting costs. The biggest cost in RAG and fine-tuning is usually engineering time, not compute.

Because providers price differently and change prices often, estimate with your own numbers: requests per day, average input and output tokens, and the provider's current rates. For ways to cut the bill whichever path you choose, see our guide to reducing LLM API costs.

How do you decide between fine-tuning and RAG?

Decide by testing in order, with an evaluation set you trust. Each step only happens if the previous one measurably falls short.

1. Write 30 to 100 evaluation cases from real inputs, with expected answers or a grading rubric. Without this, you can't tell whether any change helped.
2. Try a strong prompt with instructions, examples and any context that fits. Measure.
3. If the model lacks knowledge (wrong or missing facts, can't cite), add the documents to the prompt if they fit, or build RAG if they don't. Measure retrieval and answers separately.
4. If the model lacks behavior (format errors, inconsistent labels) after prompt work, consider fine-tuning on examples that target exactly those failures.
5. If cost or latency is the problem, try a smaller model with a better prompt, caching, or distillation by fine-tuning a small model on a large model's outputs.

Worked examples

Can you combine RAG and fine-tuning?

Yes, and mature systems often do: a model fine-tuned for format or tone answers from documents retrieved at question time. Fine-tuning handles how to answer; RAG supplies what to answer with.

Build it in that order. Get RAG plus prompting working and measured first, find the specific failures that remain, and fine-tune only for those. Anthropic's advice for agent systems applies here too: start with the simplest solution and add complexity only when it demonstrably improves results.

Next steps: pick your approach this week

• Write 30 evaluation cases from real inputs before touching anything else.
• Build the best prompt you can, including the documents if they fit, and measure.
• If facts are the gap, add retrieval; test hybrid search and reranking before tuning chunk sizes.
• If behavior is the gap, collect examples of the exact failures and consider fine-tuning.
• Check your provider's fine-tuning status and pricing before committing.
• Re-run the evaluation after every change and keep the one that wins.

If you build the pipeline and its evaluation scripts with a coding agent, Orkio can keep agents such as Claude Code working on it on an always-on cloud box while you're away.

Frequently asked questions