Fine-tuning is a last resort, not a first move
Teaching a model is a five-rung ladder. Most teams skip to the expensive rungs. You should choose the shallowest intervention that solves your problem under evaluation.
"We should just fine-tune it" is the sentence that ends productive architecture conversations. Sometimes it is correct. Often it is a costly solution to the wrong problem. Teaching a model is not one technique: it is a ladder of interventions with different costs, different guarantees, and different failure modes. You should choose the shallowest rung that solves your problem under evaluation.
The most common expensive mistake is fine-tuning to add facts that should have been retrieved. The second most common is prompt-hacking to compensate for a capability that the model simply does not have.
The ladder
Five rungs, from cheapest to deepest: prompts with zero training, few-shot examples in context, retrieval and tools at inference time, parameter-efficient adapters like LoRA, and full fine-tuning that rewrites the weights. Higher rungs cost more in data, compute, and operational burden. They also reach deeper: the behavior persists without extra context. And they are harder to roll back.
The key distinction that most teams miss is between how and what. Fine-tuning changes how a model answers. Retrieval changes what it answers with. If your failure is "it doesn't know our internal policy," that is almost always a retrieval problem. If your failure is "it can't reliably follow a five-step procedure even when given the right facts," that is a capability problem, and the ladder starts to apply.
The first question is always "what" vs "how": is this missing knowledge, or a missing capability?
This is why the LoRA rung exists at all: Hu et al. (2021) showed it can cut the number of trainable parameters by up to 10,000x and GPU memory requirements by 3x relative to full fine-tuning of GPT-3 175B, while matching or beating full fine-tuning quality on the benchmarks they tested. That cost gap is the entire argument for trying adapters before a full fine-tune.
| Rung | Best for | Do not use when |
|---|---|---|
| Prompts | Format, tone, constraints the model already supports | The capability is not there; prompts cannot install it |
| Few-shot | Complex schemas, output patterns, structural imitation | Examples are noisy or cover more than one task shape |
| RAG / tools | Fresh facts, org-specific truth, citations | Source quality is poor or retrieval is unmonitored |
| LoRA / adapters | Persistent style, consistent tool-use patterns, domain register | The knowledge changes frequently; adapters are not a database |
| Full fine-tune | Deep domain shift, default behavior everywhere | You lack eval harness; you will ship regressions invisibly |
Anti-patterns worth naming explicitly
Fine-tuning to memorize documentation is the most common waste of compute in applied AI. If the content changes weekly, baking it into weights creates an update treadmill, destroys auditability, and costs orders of magnitude more than a retrieval index update. The second anti-pattern is skipping rungs: jumping to fine-tuning before clarifying requirements, building an eval set, or checking whether the base model with better prompting already meets the bar. Training without evals is expensive guessing.
The rungs stack. The best production systems combine a fine-tuned model for style and tool-use patterns, retrieval for facts, and a short prompt for constraints. Climbing to the right rung means measuring at each step, not optimizing for the most impressive-sounding solution.
Changing facts should stay outside model weights; that's an update treadmill, not a fine-tune.
This is the same decision tree experienced practitioners reach for:
Prompting vs RAGs vs Fine-tuning: An important decision that every AI Engineer must make when building an LLM-based application. To understand what guides the decision, let's first understand the meaning of these terms. 1锔忊儯 Prompting Engineering: The prompt is the text input聽Show more
References: LoRA: Hu et al. (2021), Hugging Face PEFT, Anthropic