Fine-tuning changes how a model behaves. Retrieval changes what it knows at the moment you ask. That single distinction settles most of the argument. If your model produces the right kind of answer with the wrong facts in it, you have a knowledge problem, and retrieval is the fix. If it has the facts available and still writes them in the wrong format, the wrong tone, or the wrong length, you have a behavior problem, and fine-tuning is the fix. Fine-tuning a model on your documents so that it will remember them is the single most common and most expensive mistake in this whole area, because training compresses facts rather than storing them, and a compressed fact is one you cannot cite, update, or trust.
Key Takeaways
- Facts go in the prompt, behavior goes in the weights. Retrieval for knowledge; fine-tuning for form, tone, schema, and refusal.
- Fine-tuning does not reliably add knowledge. It makes an answer shape more likely, which is why fine-tuned models write confident text that resembles your documents.
- LoRA is the default, and QLoRA puts a 7B fine-tune on a single consumer GPU by quantizing the frozen base to 4-bit.
- The cost shapes are opposite. RAG has no setup cost and a permanent per-request cost; fine-tuning inverts that.
- The hybrid wins in production: retrieve the facts, then train on examples containing real retrieved context.
Everything below assumes Python 3.11 or newer and a CUDA GPU for the training sections. If retrieval is new to you, start with RAG explained. There is also a decision tool on this site that walks the same logic interactively.
What Fine-Tuning and RAG Actually Change
A model's output is a function of two things: the weights it was trained with, and the tokens in the context window. Fine-tuning edits the first and leaves the second alone. RAG edits the second and leaves the first alone. Every other difference between the two follows from that.
Fine-tuning runs gradient descent over examples of the behavior you want, so that behavior becomes more likely for any input. The change is permanent, global, and invisible: you cannot look at a weight and see which example put it there. Retrieval is far less clever and far more auditable. It searches a store of text for passages relevant to the question, pastes them into the prompt, and asks the model to answer from them. The weights never move. Everything people care about follows from that difference.
| Property | Fine-tuning | RAG |
|---|---|---|
| What it changes | Model weights (or adapter weights) | Prompt contents at request time |
| Update a single fact | Retrain | Edit one row |
| Citations | Not possible | Built in, by source ID |
| Per-user permissions | One model per permission set | A filter on the query |
| Controls output format | Yes, strongly | Only via instructions |
| Controls tone and register | Yes, strongly | Weakly |
| Setup cost | Data curation plus GPU time | Ingest pipeline plus a vector store |
| Per-request cost | Lower (short prompts) | Higher (context tokens every time) |
| Fails by | Confidently inventing plausible facts | Retrieving the wrong passage |
The Test: Knowledge Problem or Behavior Problem?
Take twenty real failures and sort them into two piles. Pile one: the answer contains something false, missing, or out of date. Pile two: the answer is factually fine but arrives in the wrong shape, voice, or length. Pile one is retrieval work, pile two is fine-tuning work, and the size of each tells you where the month goes.
The sorting usually ends the argument, because the piles are rarely balanced. Support assistants, internal search, and policy question-answering are almost entirely pile one. Document generation, structured extraction, ticket triage, and anything with a house style are heavily pile two.
The answer exists in a document somewhere
The model says the deductible is $500 when the policy says $750. It cites a procedure replaced in March. It cannot answer because the information was written after training. Retrieval fixes all three; training fixes none durably.
The facts are right, the output is wrong
Six paragraphs when you need four bullets. Your JSON schema ignored one time in twenty. Hedging when reviewers want a verdict. Domain vocabulary that will not hold steady. That is what training changes.
One warning about the second pile: before you fine-tune for output format, check whether constrained decoding solves it for free. Schema-constrained generation enforces valid JSON at the sampling step instead of hoping the model complies, and it removes a large share of the cases people used to train for. See the structured outputs guide.
Why Fine-Tuning Cannot Fix a Knowledge Problem
Training on documents does not install those documents into the model. It shifts the probability distribution toward text that looks like them. The model becomes fluent in your domain while still producing facts that were in no source, now in a register that makes the errors harder to catch.
Four properties make this a structural limit rather than a tuning problem.
Facts get blended, not filed
Updates spread information across parameters shared with everything else. Two similar policies become one averaged policy.
You cannot cite a weight
A reviewer asking "where did that number come from" has no answer. For regulated work, that alone rules the approach out.
Freshness requires retraining
One changed paragraph means a new run, a new evaluation, and a new deployment. Retrieval makes it an update statement.
Permissions become impossible
If users may see different documents, weights holding all of them leak across the boundary. Filters happen at query time or not at all.
One narrow exception, named honestly: continued pretraining on tens or hundreds of millions of domain tokens does improve a model's fluency and priors. It is a different activity from instruction fine-tuning, costs far more, and still gives you no citations and no freshness.
What Fine-Tuning Is Genuinely Good At
Fine-tuning earns its cost in four situations: enforcing an output format prompting cannot hold, installing a house voice, teaching narrow domain behavior such as classification or extraction, and making a small model do one job as well as a large one so you can serve it cheaply.
The last case pays for itself most often. A fine-tuned 7B model that handles one task at the quality of a much larger model turns a per-token bill into a fixed serving cost, and cuts latency at the same time, because the prompt shrinks from two thousand tokens of instructions and examples to a couple of hundred.
Distillation is the mechanism there, and it is often misunderstood. You run a large model over real inputs, keep only the outputs a human accepted, and train a small model on those pairs. The small model is not learning the world; it is learning your task. That narrow behavior transfer is exactly what fine-tuning does well.
LoRA and QLoRA, Explained Properly
Full fine-tuning updates every weight, holding parameters, gradients, and optimizer state at roughly 16 bytes per parameter with Adam in mixed precision. LoRA freezes the base and trains a small pair of low-rank matrices injected into selected projection layers, cutting trainable parameters to well under one percent. QLoRA quantizes the frozen base to 4-bit while training those adapters in higher precision, which is what puts a 7B fine-tune on a single consumer GPU.
The arithmetic explains the design. A 7-billion-parameter model in bf16 is about 14 GB of weights before you train anything. Add gradients and two Adam moment buffers and you are far past what a 24 GB card holds. Freezing the base removes both for those parameters.
LoRA works because the useful update to a weight matrix tends to be low rank. Instead of a full update for a layer of shape d_out by d_in, you learn two thin matrices, A of shape r by d_in and B of shape d_out by r, and add their product to the frozen weight. With r at 8 or 16 the parameter count collapses. The LoRA paper's headline result was a reduction in trainable parameters of several orders of magnitude with quality that held up on the tasks tested.
# pip install torch transformers peft trl datasets accelerate bitsandbytes
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import LoraConfig, get_peft_model
import torch
BASE = "Qwen/Qwen2.5-7B-Instruct"
tok = AutoTokenizer.from_pretrained(BASE)
model = AutoModelForCausalLM.from_pretrained(
BASE, torch_dtype=torch.bfloat16, device_map="auto"
)
lora = LoraConfig(
r=16, # rank: 8-16 for style/format, 32-64 for harder shifts
lora_alpha=32, # scaling; a common default is 2x the rank
lora_dropout=0.05,
bias="none",
task_type="CAUSAL_LM",
target_modules=[
"q_proj", "k_proj", "v_proj", "o_proj",
"gate_proj", "up_proj", "down_proj",
],
)
# See what the config costs before you train anything
get_peft_model(model, lora).print_trainable_parameters()
# prints trainable vs total; with this config the trainable share
# lands well under 1% of the base model.
#
# get_peft_model() injects adapters into `model` in place, and the
# trainer below applies `lora` itself. Reload the base from BASE
# before training - applying LoRA twice is an error, not a no-op.
Two knobs matter most. r is capacity: start at 16, raise it only if a measured evaluation says the adapter is underfitting. target_modules is coverage: attention projections alone often suffice for tone and format, and the MLP projections help when the task sits further from the base model's defaults.
QLoRA changes one thing. The frozen base loads in 4-bit using the NF4 data type, with a second quantization pass over the quantization constants themselves, while the adapters stay in bfloat16 and gradients flow through the dequantized weights. The QLoRA paper demonstrated fine-tuning a 65B model on a single 48 GB GPU this way.
from transformers import AutoModelForCausalLM, BitsAndBytesConfig
from peft import prepare_model_for_kbit_training
import torch
bnb = BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_quant_type="nf4", # 4-bit NormalFloat
bnb_4bit_use_double_quant=True, # quantize the quant constants too
bnb_4bit_compute_dtype=torch.bfloat16,
)
model = AutoModelForCausalLM.from_pretrained(
BASE, quantization_config=bnb, device_map="auto"
)
model = prepare_model_for_kbit_training(model)
model.config.use_cache = False # required with gradient checkpointing
model.gradient_checkpointing_enable()
Training data is a JSONL file of chat conversations, one example per line, each demonstrating exactly the behavior you are training for.
# data/train.jsonl, one object per line:
# {"messages": [{"role": "system", "content": "..."},
# {"role": "user", "content": "..."},
# {"role": "assistant", "content": "..."}]}
from datasets import load_dataset
from trl import SFTConfig, SFTTrainer
ds = load_dataset("json", data_files={
"train": "data/train.jsonl",
"test": "data/holdout.jsonl", # never seen during training
})
cfg = SFTConfig(
output_dir="out/house-format",
num_train_epochs=2, # 2-3 is plenty; more overfits fast
per_device_train_batch_size=2,
gradient_accumulation_steps=8, # effective batch of 16
learning_rate=1e-4, # LoRA tolerates 10x full-FT rates
lr_scheduler_type="cosine",
warmup_ratio=0.03,
bf16=True,
logging_steps=10,
save_strategy="epoch",
)
trainer = SFTTrainer(
model=model, # the freshly loaded base, not a PEFT wrap
args=cfg,
train_dataset=ds["train"],
eval_dataset=ds["test"],
processing_class=tok, # older TRL releases name this `tokenizer`
peft_config=lora, # TRL attaches the adapter for you
)
trainer.train()
trainer.save_model("out/house-format") # adapter only, a few dozen MB
Verify one thing your trainer may or may not be doing: masking the prompt so loss is computed only on the assistant turn. Training on the user's text too teaches the model to generate questions, which is not the job. TRL ships a completion-only collator for exactly this. Pin the library versions in a lockfile while you are here: TRL has renamed trainer arguments across releases, so a script that ran last quarter will not always run against a fresh install.
Serving is where the adapter design pays off. The output is a small adapter file rather than a new copy of the model, so one base can host several adapters at once.
# one base model, multiple adapters, hot-swapped per request
vllm serve Qwen/Qwen2.5-7B-Instruct \
--enable-lora \
--lora-modules house-format=out/house-format triage=out/triage \
--max-lora-rank 16
# then call it with the adapter name as the model:
# {"model": "house-format", "messages": [...]}
The vLLM guide covers throughput and memory settings, Ollama is the easier path for local experiments, and the Transformers guide goes deeper on the training loop.
Cost and Latency: The Real Trade
RAG costs nothing up front and something on every request, because retrieved context is input tokens and retrieval plus reranking is milliseconds. Fine-tuning costs real money once and then makes every request cheaper and faster. The break-even point is a function of request volume and how often the task changes.
Here is the calculation. Substitute your provider's published prices and your GPU hourly rate; the placeholders below are round numbers chosen to be obviously illustrative.
# PLACEHOLDER prices - substitute your provider's real published rates
PRICE_IN = 1.00 / 1_000_000 # dollars per input token
GPU_HOUR = 2.00 # dollars per GPU-hour
# Measure these two on your own app, do not guess them
RAG_PROMPT_TOKENS = 2500 # instructions + 5 retrieved chunks
FT_PROMPT_TOKENS = 300 # short prompt, behavior is in the weights
REQUESTS_PER_MONTH = 100_000
TRAIN_GPU_HOURS = 6 # one LoRA run on a single card
RETRAINS_PER_YEAR = 4 # base model bumps, spec changes, drift
saved_tokens = (RAG_PROMPT_TOKENS - FT_PROMPT_TOKENS) * REQUESTS_PER_MONTH
monthly_saving = saved_tokens * PRICE_IN
annual_training = TRAIN_GPU_HOURS * GPU_HOUR * RETRAINS_PER_YEAR
print(monthly_saving * 12, annual_training)
# The comparison is only meaningful if the fine-tuned model
# scores at least as well on your held-out set. Check that first.
Two costs get left out of these spreadsheets and both exceed the GPU bill. Data curation: writing and reviewing several hundred consistent examples is days of skilled work, not hours. And the retraining treadmill, because a fine-tuned model is pinned to one base checkpoint, so every base upgrade means running the pipeline again.
On latency the ordering is clear. Retrieval adds a vector search, often a lexical search, sometimes a cross-encoder rerank, then a much longer prompt to read before the first token. A short-prompt fine-tuned model skips all of it. Where time-to-first-token is a product requirement, that gap matters more than the token bill.
Prompting vs RAG vs Fine-Tuning, Compared Honestly
Prompting is the correct first attempt for almost everything, and many projects never need more. RAG is correct when the knowledge is large, changing, permissioned, or must be cited. Fine-tuning is correct when the behavior is stable and prompting cannot hold it. The three are ordered by cost, and skipping to the expensive one is the usual mistake.
| Question | Prompting | RAG | Fine-tuning | Hybrid |
|---|---|---|---|---|
| Fixes wrong facts | Only if you paste them | Yes | No | Yes |
| Fixes wrong format | Often | No | Yes | Yes |
| Handles a changing corpus | Manually | Yes | No | Yes |
| Produces citations | No | Yes | No | Yes |
| Per-user permissions | No | Yes | No | Yes |
| Setup effort | Hours | Days to weeks | Weeks | Weeks |
| Per-request cost | Low | High | Lowest | Moderate |
| Added latency | None | Retrieval plus long prompt | None | Retrieval |
| Needs a GPU | No | No | Yes | Yes |
| Needs an eval set | Yes | Yes | Yes | Yes |
The last row is the one nobody wants. Every method above is unmeasurable without a held-out set of real questions with known-good answers, and teams that skip it choose by vibe. The LLM evaluation guide covers building one small enough to write in a day and strict enough to fail a bad change.
The Hybrid Pattern Most Production Systems Use
Retrieval supplies the facts, fine-tuning teaches the model what to do with them. The detail that makes it work: training examples must contain real retrieved context from the same retriever you ship, so the model learns on the distribution it will see at inference rather than on clean hand-written passages.
Train on clean context, deploy against messy retrieval, and you get a model that has never seen a near-miss passage and no idea it should ignore one. Train on the retriever's actual output, including the mediocre chunks that always come back at rank four and five, and the model learns to use what is useful, ignore what is not, and refuse when nothing answers the question. That refusal behavior is the most valuable thing fine-tuning contributes to a retrieval system.
# Build SFT examples whose prompts contain REAL retrieved context
import json
HOUSE_RULES = (
"Answer only from the numbered sources below. "
"Cite every claim as [doc_id#chunk_id]. "
"If the sources do not contain the answer, say so and stop."
)
with open("data/train.jsonl", "w") as out:
for question, gold_answer in golden_set:
chunks = retrieve(question, k=5) # the retriever you SHIP
context = "\n\n".join(
f"[{c.doc_id}#{c.chunk_id}] {c.content}" for c in chunks
)
record = {"messages": [
{"role": "system", "content": HOUSE_RULES},
{"role": "user", "content": f"{context}\n\nQuestion: {question}"},
{"role": "assistant", "content": gold_answer}, # cited, house format
]}
out.write(json.dumps(record) + "\n")
# Include 10-15% examples where retrieval returns NOTHING useful
# and the gold answer is a refusal. This is what teaches the model
# to stop inventing when the context is thin.
Build the retrieval half first and measure it alone. If recall at 50 is poor, no fine-tune rescues the answer, because the correct passage was never in the prompt. The RAG pipeline guide covers chunking, hybrid search, and reranking; the vector database comparison covers where the vectors live.
The Failure Modes Beginners Hit
Most first fine-tunes fail for one of six reasons, and none of them are the learning rate: chat template mismatch, unmasked prompt loss, too few or inconsistent examples, no held-out set, catastrophic forgetting, and a serving stack that quietly loads the wrong base checkpoint.
Two more from code review: training on synthetic data the model generated itself, which teaches it to imitate its own errors more confidently, and not versioning the training set, so a good run from six weeks ago cannot be reproduced.
When to Use Neither
If the corpus fits in the context window, put it in the prompt and turn on caching. If the question is an aggregate rather than a lookup, write SQL. If three examples in the system prompt fix the format, you are done. Each has a fraction of the moving parts and none of the maintenance.
Small, stable, or already solved by a prompt
A forty-page handbook belongs in a cached system prompt. "How many contracts closed last quarter" is a database query; similarity search returns some rows, never a correct count. A format that three examples fix does not need a GPU.
Large and changing, or a behavior prompting cannot hold
Corpus far bigger than the window, changing weekly, differing by user, or requiring citations: build retrieval. Format or voice that slips no matter how the prompt is written, at volume: fine-tune.
Long context windows genuinely shrank the set of problems that need a retrieval pipeline. They did not touch the ones defined by permissions, freshness, or citations, which are the ones that matter most in regulated work.
How to Decide in One Afternoon
Write fifty real questions with known-good answers, run your current prompt against them, and sort every failure into one of five buckets. The bucket counts pick the method for you. This takes an afternoon and replaces weeks of arguing.
# Run your current prompt over 50 real questions, then label each failure.
# The bucket with the most cases is where the work goes.
from collections import Counter
def triage(case):
if not case.answer_exists_in_corpus:
return "NEITHER: the fact does not exist yet. Write the document."
if not case.correct_chunk_retrieved:
return "RETRIEVAL: chunking, hybrid search, reranking."
if case.correct_chunk_retrieved and case.facts_wrong:
return "CONTEXT ASSEMBLY: budget, ordering, or the answer prompt."
if case.facts_right and case.form_wrong:
return "FINE-TUNE: format, tone, schema, refusal behavior."
return "PASSING"
counts = Counter(triage(c) for c in eval_set)
# Mostly RETRIEVAL -> build the RAG pipeline, do not train anything
# Mostly FINE-TUNE -> curate 300 examples, run one LoRA, re-measure
# Both, roughly evenly -> retrieval first, then fine-tune on its output
# Mostly NEITHER -> this is a documentation problem, not an AI one
Run retrieval work first when both buckets are full. Retrieval is cheaper, reversible, and produces the context that the fine-tuning dataset will need anyway. Training before the retriever is stable means building a dataset you will have to throw away.
Frequently Asked Questions
Does fine-tuning teach a model new facts?
Not reliably, and not in a way you can audit. Training makes the shape of an answer more likely, and specific facts get compressed and blended with everything else in the training set. A fine-tuned model often produces something that looks like your document rather than what your document says, with no way to tell which happened. Facts that change strand you as well: one revised policy paragraph means retraining, where a retrieval system means editing one row.
Is LoRA as good as full fine-tuning?
For the tasks most teams actually fine-tune for, yes. LoRA freezes the base weights and trains a small pair of low-rank matrices added to selected projection layers, keeping trainable parameters under one percent of the model. That is enough capacity for format, tone, schema adherence, and narrow domain behavior. Full fine-tuning keeps an edge when you are teaching a genuinely new capability, but that is a research budget rather than a product decision.
How many examples do I need to fine-tune a model?
For a format or style task, a few hundred well-constructed examples usually move the metric more than several thousand scraped ones. Every example should demonstrate exactly the behavior you want, with the same structure, citation style, and refusal behavior. Hold out at least fifty the model never sees. If you cannot write two hundred examples that agree with each other, the specification is not clear enough to train against yet.
Which is cheaper, RAG or fine-tuning?
The cost shapes are opposite. RAG has almost no setup cost and a permanent per-request cost, because every request carries retrieved context as extra input tokens. Fine-tuning costs real money once and then makes every request cheaper, because the behavior lives in the weights instead of a long prompt. It also carries a recurring cost people forget: retraining whenever the base model or the task changes. At low volume RAG usually wins; at high volume with a stable task, a fine-tuned smaller model often wins.
Can I use RAG and fine-tuning together?
Yes, and that is what most mature production systems run. Retrieval supplies the facts at request time so answers stay fresh and citable, and fine-tuning teaches the model how to use retrieved context: which parts to trust, how to cite them, what format to emit, and when to refuse. The detail that matters is that training examples must contain real retrieved context from the same retriever you ship.
Further reading: LoRA: Low-Rank Adaptation of Large Language Models, QLoRA: Efficient Finetuning of Quantized LLMs, Hugging Face PEFT documentation, TRL supervised fine-tuning docs