RAG or fine-tuning: which should you use?

The conversation usually starts with “let's train a model on the company's PDFs.” In practice, what the company wants is an answer with the current document, with permission, and with the ability to update the file tomorrow. That describes RAG. Fine-tuning is a different tool.

The problem

A generic model invents policy. A model “trained” the wrong way memorizes a stale excerpt and becomes expensive to update. Without a clear difference, the project spends budget in the wrong place and still cannot say which file the answer came from.

How each one works

RAG (retrieval-augmented generation) retrieves authorized excerpts from the knowledge base — documents, procedures, catalogs — and delivers that context to the model at question time. The model stays generic. What changes is the material it can use in that answer.

Fine-tuning adjusts a model's weights using examples. It can improve response format, style, or performance on repeated tasks with many stable examples. It is not an efficient way to keep a model current with frequently updated legal documents.

When to use RAG

• Policies, procedures, and manuals that change.
• Customer service that needs to cite the current rule.
• Several departments with different knowledge bases and different permissions.
• A need to know which document fed the answer.

When fine-tuning can enter

Fine-tuning can help with stable classification tasks, strict output formats, or narrow domains supported by thousands of curated examples. Teams often use fine-tuning for behavior and RAG for current factual information.

Main RAG challenges

A document with no owner. An entire folder indexed with no cut. A sales agent reading an HR file. An index that does not update when the PDF changes. A question that is actually an order status — and should go to the API, not to the semantic index.

Common mistakes

• Fine-tuning because the term sounds more advanced.
• Dumping the entire drive into the index and calling that governance.
• Having no “I don't know, I'll transfer” answer.
• Measuring success only by “it sounds smart” instead of accuracy on real policy.

Practical experience

In platforms with multiple agents and knowledge bases, each agent needs defined document access, clear ownership, and usage monitoring. RAG without access controls can expose information; fine-tuning without curated data produces unreliable and outdated responses.