RAG vs Fine-Tuning: How to Choose

The question behind the question

"Should we use RAG or fine-tuning?" is usually the wrong framing, because the two techniques solve different problems. Retrieval-augmented generation changes what the model knows at the moment it answers. Fine-tuning changes how the model behaves across every answer it produces.

Once you internalise that distinction, most architecture debates resolve themselves. If your failure mode is the model not knowing a fact, retrieval is the fix. If your failure mode is the model knowing the fact but presenting it badly, fine-tuning is the fix. If both are failing, you need both, and you should build the retrieval layer first.

How RAG actually works

Retrieval-augmented generation inserts a search step before generation. At query time the system embeds the user's question, searches a vector index for semantically similar chunks of your content, and places the best matches into the prompt as context. The model then answers using that context rather than relying purely on parameters learned during training.

The pipeline has four stages, and the quality of the final answer is capped by the weakest one.

1. Chunking. Source documents are split into passages. Too large and retrieval becomes imprecise; too small and passages lose the context needed to be meaningful.
2. Embedding. Each chunk is converted into a vector that encodes its meaning, then stored in a vector database.
3. Retrieval. The query is embedded and the nearest chunks are returned, often combined with keyword search in a hybrid approach.
4. Generation. Retrieved passages are inserted into the prompt with instructions to answer from the provided context and to say so when the answer is absent.

The dominant advantage is freshness. Update a document, re-index it, and the system is current within minutes. No retraining, no model deployment, no evaluation cycle. For any domain where facts change — pricing, policies, documentation, inventory, regulations — this is decisive.

The second advantage is attribution. Because you know which chunks were retrieved, you can cite sources. In regulated, enterprise or support contexts this is frequently a hard requirement rather than a nice feature.

Where RAG systems actually fail

Teams often conclude that RAG does not work when what they have built is a poor retrieval system attached to a capable model. The generation step is rarely the problem.

• Chunking that ignores document structure. Splitting on a fixed character count cuts tables in half and separates headings from the content they describe. Split on semantic boundaries — sections, headings, paragraphs — and keep a small overlap.
• Pure vector search for keyword-shaped queries. Embeddings capture meaning but handle exact identifiers, error codes and product names poorly. Hybrid search combining BM25 keyword matching with vector similarity fixes a large class of failures cheaply.
• No reranking. Initial retrieval optimises for recall. A cross-encoder reranker applied to the top twenty results substantially improves precision, and precision is what the model consumes.
• Stuffing too much context. More retrieved passages is not better. Irrelevant context dilutes attention and measurably increases hallucination. Retrieve broadly, rerank, then pass only the best three to five passages.
• Stale or duplicated indexes. Near-duplicate documents crowd out diverse results, and an index that drifts from the source is worse than no retrieval at all because it looks authoritative.

A practical rule: before blaming the model, print the retrieved chunks for twenty failing queries. In most audits, the correct passage was never retrieved, which means no amount of prompt engineering could have saved the answer.

What fine-tuning is genuinely good at

Fine-tuning continues training on your examples, adjusting weights so the model internalises patterns of behaviour. Modern practice overwhelmingly uses parameter-efficient methods such as LoRA, which train a small set of adapter weights rather than the full network, reducing cost and hardware requirements by orders of magnitude.

Fine-tuning excels at:

• Consistent output format. If you need reliable JSON conforming to a specific schema, or a consistent document structure, training on a few hundred examples outperforms increasingly elaborate prompts.
• Tone and voice. Matching a brand's writing style, a clinical register, or a particular level of formality is a behavioural property, not a factual one.
• Domain-specific language. Specialist jargon, internal terminology and unusual abbreviations become natural rather than something the model reasons around.
• Classification and extraction. Narrow, well-defined tasks with abundant labelled examples often reach higher accuracy with a fine-tuned smaller model than with prompting a larger one.
• Latency and cost reduction. A fine-tuned small model that no longer needs a lengthy system prompt with examples can be dramatically cheaper per request at scale.

What fine-tuning is not good at is reliably teaching facts. Training on documents does not create a lookup table; it shifts probability distributions. The model becomes more likely to produce text resembling your documents, which is subtly different from knowing their contents — and when it is wrong, it is wrong fluently and without citation.

The cost and effort comparison

The two approaches distribute effort very differently across time.

RAG has a moderate initial build — ingestion, chunking, embedding, a vector store, retrieval logic and evaluation — and then ongoing per-query costs, because every request pays for embedding, retrieval and a longer prompt. Maintenance is mostly data hygiene: keeping the index synchronised with the source.

Fine-tuning front-loads the effort into dataset construction, which is the part teams consistently underestimate. Producing several hundred high-quality, consistent examples is slow, and quality matters far more than quantity. Training itself is comparatively cheap with LoRA. Inference is then cheaper per request. The recurring cost is retraining whenever behaviour needs to change.

A useful heuristic: RAG cost scales with query volume, fine-tuning cost scales with change frequency.

A decision framework

Choose RAG when:

• The information changes regularly
• You must cite sources or show provenance
• The knowledge base is large or continuously growing
• Different users should see different subsets of data, enforced by access control at retrieval time
• You need to add or remove knowledge quickly, including for compliance deletion requests

Choose fine-tuning when:

• Output structure or format must be highly consistent
• Tone, voice or style is central to the product
• The task is narrow, repetitive and well-represented by examples
• Prompt length has become a meaningful cost or latency problem
• You want a smaller model to match a larger one on a specific task

Use both when you need a system that speaks in a consistent, domain-appropriate voice about facts that change — which describes most serious production assistants. The fine-tune governs behaviour; retrieval supplies truth.

Start with prompting, and be honest about it

Before either approach, exhaust prompt engineering. A carefully constructed system prompt with three or four well-chosen examples solves a surprising proportion of problems at zero infrastructure cost, and it establishes the baseline you will measure everything else against.

The recommended progression is deliberately boring:

1. Prompt engineering with few-shot examples
2. Add retrieval when the failure mode is missing knowledge
3. Improve retrieval quality — hybrid search, reranking, better chunking
4. Fine-tune only when the remaining failures are behavioural

Most teams that jump straight to fine-tuning discover afterwards that their real problem was retrieval, and they have now built an expensive artefact that must be maintained.

Evaluate, or you are guessing

An end-to-end score tells you the system is wrong without telling you which half to fix.

Whichever path you take, the deciding factor is measurement. Build an evaluation set of at least a hundred representative queries with known-good answers before you build the system, not after.

Track retrieval and generation separately:

• Retrieval metrics: recall at k, mean reciprocal rank, and the proportion of queries where the correct passage appeared at all.
• Generation metrics: factual accuracy against the source, citation correctness, refusal rate on unanswerable questions, and format compliance.

Separating these is essential, because an end-to-end score tells you the system is wrong without telling you which half to fix. Teams that measure both consistently converge on working systems far faster than teams iterating on intuition.

If you are running local models while experimenting, the hardware constraints are covered in running a local LLM on 8GB of VRAM.

Frequently asked questions