← AI for Professionals: Using LLMs at Work
Lesson
Lesson 1: Grounding the model in your own data (RAG)
Explain retrieval-augmented generation as grounding the model in external documents and citing retrieved passages to reduce hallucinations.
What is RAG and why does it matter?
Retrieval-Augmented Generation (RAG)
A standard LLM answers questions purely from patterns learned during training. It does not access your internal documents, company wiki, or any data that wasn't part of that training corpus. Ask it about your company's latest refund policy and it will either guess, say it doesn't know, or — worse — confidently invent an answer that sounds plausible but is wrong. This is called a hallucination.
RAG (Retrieval-Augmented Generation) solves this by adding a retrieval step before the model responds. The system first searches a knowledge base — your documents, database, or uploaded files — and pulls out the passages most relevant to the question. Those passages are then inserted into the prompt as context, and the model generates its answer grounded in that retrieved text, typically citing the source chunks.
This approach significantly cuts hallucinations on your own data because the model is no longer guessing — it is paraphrasing and referencing concrete text you provided. It also makes answers checkable: you can open the cited passage and verify the claim yourself.
Critically, RAG reduces but does not eliminate hallucinations. The model can still misread, misquote, or selectively emphasize retrieved text. If the retrieval step returns the wrong passages, the model will confidently generate an answer based on irrelevant or misleading content. Always verify important claims against the source, even with RAG in place.
Lesson notes
Retrieval-Augmented Generation (RAG)
A standard LLM answers questions purely from patterns learned during training. It does not access your internal documents, company wiki, or any data that wasn't part of that training corpus. Ask it about your company's latest refund policy and it will either guess, say it doesn't know, or — worse — confidently invent an answer that sounds plausible but is wrong. This is called a hallucination.
RAG (Retrieval-Augmented Generation) solves this by adding a retrieval step before the model responds. The system first searches a knowledge base — your documents, database, or uploaded files — and pulls out the passages most relevant to the question. Those passages are then inserted into the prompt as context, and the model generates its answer grounded in that retrieved text, typically citing the source chunks.
This approach significantly cuts hallucinations on your own data because the model is no longer guessing — it is paraphrasing and referencing concrete text you provided. It also makes answers checkable: you can open the cited passage and verify the claim yourself.
Critically, RAG reduces but does not eliminate hallucinations. The model can still misread, misquote, or selectively emphasize retrieved text. If the retrieval step returns the wrong passages, the model will confidently generate an answer based on irrelevant or misleading content. Always verify important claims against the source, even with RAG in place.