Prompting vs RAG vs fine-tuning: choosing

The big idea

When a model does not do what you want, there are three levers. Prompting: tell it more clearly, with instructions and examples. Retrieval (RAGRetrieval-augmented generation: fetch documents relevant to a query from your own data and put them in the prompt so the model answers from them.): give it the facts it is missing, fetched at question time. Fine-tuning: change its weights so the behaviour becomes a habit.

Picking the wrong lever is one of the most expensive mistakes in AI projects. Teams fine-tune a model on their product manuals and find it still gets details wrong and goes stale with the next release. Others build a retrieval pipeline when the real problem was output format. The question that sorts most cases is simple: is this a knowledge problem or a behaviour problem?

Think of a new employee. If they keep getting the prices wrong, you do not send them on a training course; you give them the price list (retrieval). If they have the price list but write rambling, badly structured replies to customers, a better brief helps first (prompting), and if that is not enough, coaching with many examples builds the habit (fine-tuning).

How it works

Start with prompting

Prompting is the cheapest experiment: minutes to try, nothing to train or host. Clear instructions, a few examples ( Few-shot prompting and examples ), structured output settings ( Structured outputs ) and a well-chosen model solve a surprising share of problems. You also need a small evaluation set to know whether any change helped ( Building an eval set ), and that set is reused by every later step.

Knowledge problems: retrieval

If the model lacks facts, because they are private (your tickets and contracts), newer than its training data, or niche, put them in the context at question time. Retrieval wins on:

• Freshness: update a document and the next answer uses it. No retraining.
• Traceability: answers can cite the passages they used, so people can check them.
• Access control: retrieve only what the user is allowed to see.

Fine-tuning is a poor way to add facts. The model learns them unreliably, may blend them with what it already believed, cannot tell you where an answer came from, and must be retrained whenever the facts change.

Behaviour problems: fine-tuning

If the model has the knowledge but will not consistently produce the format, tone, or narrow skill you need, and careful prompting with examples has not fixed it, fine-tuning turns the behaviour into a default. Good fits: a strict output schema at scale, a house writing style, a classification or extraction task with clear labels, a domain's terse jargon. You need examples, typically hundreds to thousands of good ones, and an evaluation set.

Cost and latency: fine-tuning a smaller model

A third reason to fine-tune is economics. If a frontier model does the task well but is too slow or expensive at your volume, fine-tuning a smaller model on examples of the big model's good outputs can match it on that narrow task at a fraction of the cost ( Distillation and synthetic data , Routing, cascades and cost optimization ). Long few-shot prompts also cost tokens on every call; baking the behaviour in removes them.

Combining them

These are not exclusive. A support assistant might use a fine-tuned small model for the house tone and output format, retrieval for product facts, and a system prompt for the rules of the day. Add each piece only when an evaluation shows it is needed.

Try it

Try this:

1. A legal team wants answers grounded in their own contract library, with the clause cited. Answer yes to questions 1 and 2. Retrieval lights up; fine-tuning does not.
2. An app must turn messy emails into a strict JSON schema millions of times a day; a large model does it well but costs too much, and you have logged 20,000 good examples. Answer yes to 3, 4 and 5. Fine-tuning lights up.
3. Same as 2, but with no examples yet. The tool tells you to prompt with few-shot examples and collect data first: you cannot fine-tune what you cannot show.

Going deeper

Long context as a fourth option

Large context windows let you paste a whole manual into the prompt. For a small, stable document set this can replace retrieval: simpler, and the model sees everything. It costs tokens on every call (prompt caching helps, Prompt caching and cost control ), adds latency, and models use the middle of long contexts less reliably ( Context windows and the KV cache ). For large or changing corpora, retrieval scales better.

Hidden costs of fine-tuning

Fine-tuning is not a one-off. You need a data pipeline, training runs, evaluations, a way to host the tuned model, and a plan to redo it when the base model is upgraded or your requirements shift. Prompts and retrieval indexes move to a new base model much more easily.

Volume decides whether that work pays off. A rough daily bill, with made-up round prices you should replace with your own:

# Daily cost: a big model with a long few-shot prompt vs a fine-tuned small model.
# Prices are made-up round numbers (dollars per million tokens); plug in your own.
BIG = {"in": 3.00, "out": 15.00}
SMALL = {"in": 0.20, "out": 0.80}

calls_per_day = 1_000_000
task_in, task_out = 300, 200     # tokens: the email in, the JSON out
few_shot = 2_000                 # tokens of instructions and examples per call

def daily(price, tokens_in, tokens_out):
    per_call = (tokens_in * price["in"] + tokens_out * price["out"]) / 1e6
    return per_call * calls_per_day

big_prompted = daily(BIG, few_shot + task_in, task_out)
small_tuned = daily(SMALL, task_in, task_out)   # behaviour is in the weights
print(f"big model + few-shot prompt: ${big_prompted:>9,.0f} per day")
print(f"fine-tuned small model:      ${small_tuned:>9,.0f} per day")

one_off = 20_000   # assumed: data prep, training runs, evals for one fine-tune
saving = big_prompted - small_tuned
print(f"one-off fine-tuning cost pays back in {one_off / saving:.1f} days")

for calls in (1_000, 10_000, 100_000):
    s = saving * calls / calls_per_day
    print(f"at {calls:>7,} calls/day: payback {one_off / s:>6.0f} days")

big model + few-shot prompt: $    9,900 per day
fine-tuned small model:      $      220 per day
one-off fine-tuning cost pays back in 2.1 days
at   1,000 calls/day: payback   2066 days
at  10,000 calls/day: payback    207 days
at 100,000 calls/day: payback     21 days

At a million calls a day, dropping 2,000 tokens of examples and moving to a small model cuts the bill from about 9,900 to 220 dollars a day, so an assumed 20,000 dollars of fine-tuning work pays back in about two days. At 1,000 calls a day the same work takes over five years to pay back: keep prompting.

When the answer is "none of these"

Some failures are not fixed by any lever: tasks that need exact computation (use a tool, Function calling and tool use ), problems that need a stronger model, or requirements that are ambiguous to begin with. Error analysis ( Error analysis ) tells you which kind of failure you have before you spend money on a fix.

Trade-offs

Common mistakes

• Fine-tuning on documents to "teach the model our knowledge base". Use retrieval for facts.
• Building RAG when the problem was the output format or tone.
• Skipping the evaluation set, so nobody can tell whether a change helped.
• Fine-tuning before trying a good prompt with examples, the cheapest experiment.
• Treating the choices as exclusive. Mature systems often combine all three.

In the wild

Where you will see this: customer support assistants commonly pair retrieval over help-centre articles with a system prompt for policy; coding assistants retrieve relevant files from your repository rather than training on it; and high-volume extraction or classification pipelines often fine-tune a small model once a larger one has shown the task is feasible and produced enough good examples.

Check yourself

Recap