Explain why an LLM can produce fluent, confident statements that are factually wrong, including the 'rewarded for guessing over admitting uncertainty' framing.
Hallucination: confident and wrong
Why LLMs make things up — and sound sure about it
Hallucination is the term for when an LLM produces output that sounds authoritative and well-written but is factually wrong. This is not random noise — the text is fluent, specific, and often structured like a credible source. Common examples include invented statistics, citations to papers or articles that do not exist, and confident answers to questions the model simply does not know.
Why does this happen? Researchers at OpenAI and elsewhere argue that the core cause is what training and benchmarks reward. An LLM is evaluated on how plausible and helpful its answers sound. If the model says 'I'm not sure' it scores poorly; if it generates a confident-sounding answer — even a wrong one — it often scores better. Over millions of training examples this shapes a system that guesses rather than admits uncertainty, much like a student who fills in every blank on an exam rather than leaving questions blank.
The critical practical lesson is that fluency and confidence are not evidence of correctness. A hallucinated citation looks exactly like a real one. A made-up statistic is formatted identically to a real one. The only reliable protection is to verify: look up the claim in a primary source, check the number, confirm the paper exists. Trust the output as a useful draft or starting point, not as a finished fact.
Lesson notes
Why LLMs make things up — and sound sure about it
Hallucination is the term for when an LLM produces output that sounds authoritative and well-written but is factually wrong. This is not random noise — the text is fluent, specific, and often structured like a credible source. Common examples include invented statistics, citations to papers or articles that do not exist, and confident answers to questions the model simply does not know.
Why does this happen? Researchers at OpenAI and elsewhere argue that the core cause is what training and benchmarks reward. An LLM is evaluated on how plausible and helpful its answers sound. If the model says 'I'm not sure' it scores poorly; if it generates a confident-sounding answer — even a wrong one — it often scores better. Over millions of training examples this shapes a system that guesses rather than admits uncertainty, much like a student who fills in every blank on an exam rather than leaving questions blank.
The critical practical lesson is that fluency and confidence are not evidence of correctness. A hallucinated citation looks exactly like a real one. A made-up statistic is formatted identically to a real one. The only reliable protection is to verify: look up the claim in a primary source, check the number, confirm the paper exists. Trust the output as a useful draft or starting point, not as a finished fact.