Fine-Tuning vs RAG vs Prompting: How to Decide Before You Spend Money

A surprising number of AI projects we get asked about start with the solution already chosen. "We want to fine-tune a model on our documents." Sometimes that's exactly right. Often it's an expensive way to solve a problem that a retrieval pipeline would have handled in a week, or that better instructions would have fixed in an afternoon.

The three approaches aren't competitors so much as different tools for different failures. The trick is working out which failure you actually have.

Start by writing down what's going wrong

Before choosing anything, collect twenty or thirty real examples where the current model gets it wrong, and look at them honestly. They usually fall into one of three buckets.

The model doesn't know something. It gives an outdated price, invents a policy, can't answer questions about your product. This is a knowledge problem.

The model knows enough but behaves wrong. Wrong format, wrong tone, too verbose, ignores a rule, mishandles a task type it rarely saw in training. This is a behavior problem.

The model is fine but too slow or too expensive. You're paying a frontier model to do something a much smaller one could do, if it were taught how. This is a cost problem.

Most use cases have a mix. Which bucket dominates tells you where to start.

Prompting: try this first, always

It's cheap, it's fast, and it gets underestimated. A clear system prompt with a handful of good examples solves more behavior problems than people expect, especially with current models. If you haven't spent a day iterating on prompts against those failure examples, do that before anything else. You'll learn what the model can and can't do, and that makes every later decision cheaper.

Prompting runs out when the instructions get so long the model stops following all of them, or when the behavior you need is hard to describe but easy to show with thousands of examples.

Retrieval: for facts that live somewhere else

If the problem is knowledge, and especially knowledge that changes, retrieval is almost always the right answer. The model looks up the relevant documents at question time and answers from them. Update a document and the answer changes the same day. No retraining.

Fine-tuning is a bad way to teach facts. The model absorbs them unreliably, can't tell you where they came from, and has to be retrained every time something changes. It's a common story: a team fine-tunes on a product catalogue, then finds the model confidently quoting last quarter's prices.

Retrieval has its own failure modes, mostly around search quality: the right document exists but doesn't get found. That's worth fixing before blaming the model.

Fine-tuning: for behavior, format, and cost

Fine-tuning earns its cost in a few situations:

A consistent output structure that prompting keeps getting almost right, like extraction into a fixed schema or classification into your own categories. A domain with its own vocabulary and conventions, where base models write like outsiders. A language or dialect the base model handles poorly, including low-resource languages. And cost reduction, where a small fine-tuned model replaces a large general one on a narrow, high-volume task.

That last one is underrated. A small open-weight model, fine-tuned on a few thousand good examples, can match a much bigger model on a single task at a fraction of the running cost, on hardware you control.

The part that decides whether fine-tuning works

It's the data. Not the model, not the hyperparameters. A fine-tune learns exactly what its examples teach, including their mistakes and inconsistencies. A few thousand clean, consistent, representative examples beat a hundred thousand scraped ones nearly every time.

That's also why the evaluation set should exist before training starts: a few hundred held-out examples that reflect real use, scored the same way every time. Without it, "the new model seems better" is just a feeling. We wrote more about this in what makes a good fine-tuning dataset.

A rough decision order

Try prompting against real failure examples. If the gap is missing or changing knowledge, add retrieval. If the gap is behavior, format, language or cost, and you can get or build a few thousand good examples, fine-tune. Plenty of production systems end up using all three: a fine-tuned model, a retrieval layer for current facts, and a system prompt on top.

If you're weighing this for a real project, we do the dataset and fine-tuning work end to end, and the first thing we'll do is check whether you need it at all.