What is retrieval-augmented generation?
Fetching the right documents before the model answers, so it draws on a business's own facts.
Retrieval-augmented generation — Retrieval-augmented generation (RAG) is a technique that fetches relevant documents at query time and passes them to a language model, so answers are grounded in a specific source rather than in training data. It is used wherever the model must reflect a business's own facts.
Also known as: RAG
Why it matters
A language model's training data does not include a specific business's offers, pricing rules or playbooks. Asked a question about them directly, it either says it does not know or, worse, guesses in a confident tone. RAG fixes this at the point of the question, not through retraining: when a query comes in, the system first searches a store of the business's own documents for the most relevant passages, then hands those passages to the model along with the question, so the answer is built from real content rather than the model's general knowledge.
The technique reduces hallucination when it works, because the model has something true to ground its answer in, but it does not remove hallucination altogether. If the search step misses the right passage, because the wording did not match well or the document was never indexed, the model still produces an answer, just without the grounding that made RAG useful in the first place. That failure mode looks identical to a correct answer from the outside, which is why RAG is a mitigation, not a guarantee, and answers touching anything that matters still deserve a check.
How RAG answers a question
- 1
A question comes in
From a person or from another part of an agentic workflow.
- 2
The system searches the document store
Using semantic search to find passages relevant to the question.
- 3
Matching passages are retrieved
The most relevant few, not the whole document store.
- 4
They are passed to the model with the question
As context the model reads before answering.
- 5
The model answers grounded in that context
Ideally citing which passage the answer came from.
The mistake to watch for
Questions
- How is RAG different from fine-tuning?
- Fine-tuning changes the model's own weights using a training run, a slow and expensive process. RAG changes nothing about the model; it retrieves relevant documents at the moment of each question and hands them over as context.
- Does RAG need a special kind of database?
- Usually a store that supports semantic search, comparing meaning rather than exact words, so a question phrased differently from the source document still finds the right passage.
- Can RAG still produce a wrong answer?
- Yes. If the retrieval step fails to find the relevant passage, or the source document itself is outdated or wrong, the model's answer will be too, even though the process looks the same from outside.