Confidently-asserted but factually incorrect output produced by an AI model — including fabricated citations, invented people or events, and confabulated numerical values — that the model cannot reliably distinguish from correct output at generation time.
Definition & scope
Hallucination, in the foundation-model-output sense, was named by Ji et al. (2023, 'Survey of Hallucination in Natural Language Generation') and has become the canonical term for LLM factual error. The phenomenon decomposes into intrinsic hallucination (output contradicts available context) and extrinsic hallucination (output asserts facts that aren't grounded in context). NIST AI RMF GenAI Profile (NIST AI 600-1) names 'Confabulation' as a primary risk category, capturing the same phenomenon under a different label (NIST's choice signals a preference against anthropomorphic framing). Governance relevance touches four surfaces. (a) Liability — when an AI-mediated legal brief contains hallucinated citations (Mata v. Avianca, 2023, S.D.N.Y.), who bears responsibility: the lawyer, the AI provider, or the AI deployer? EU AI Act Art. 13 transparency requirements + Art. 86 right-to-explanation are the closest binding frame. (b) Disclosure — should providers disclose hallucination rates as part of model-card disclosures (EU AIA Art. 53)? Industry practice is partial. (c) Redress — when hallucinated output causes harm (defamation via fabricated facts, financial loss via wrong numbers), redress mechanisms are unclear. EU AIA Art. 85 + OECD Principle 1.5 (accountability) frame the obligation; operationalisation is inconsistent. (d) Sectoral safety — hallucination in healthcare (medical-misinformation), criminal-justice (false-positive risk scores), and education (factual errors as authoritative output) drives most sectoral guidance. NIST AI 600-1 explicitly treats confabulation as a primary risk; UK AISI evaluations include factuality probes; Brazil PL 2338/2023 includes accuracy obligations. Methodologically, hallucination cannot be eliminated by current architectures (Xu et al. 2024, 'Hallucination is Inevitable'). Mitigation is via retrieval-augmented generation, confidence calibration, and post-hoc verification — not architectural fixes.
Mechanism and Typology
Hallucination is not random noise but a structural product of how autoregressive models generate text: they sample the most probable next token rather than retrieving a verified fact, so a fluent but ungrounded continuation is mechanically indistinguishable from a correct one at generation time. The canonical decomposition 1 separates intrinsic hallucination, where output contradicts the supplied context, from extrinsic hallucination, where output asserts facts absent from any context. A more recent reframing locates the root cause in training and evaluation incentives that reward confident guessing over admitting uncertainty 2. Critically, Xu et al. 3 argue the phenomenon cannot be eliminated by current architectures, recasting it as a property to be managed rather than a defect to be patched.
Governance Surfaces and Engaging Provisions
Four governance surfaces engage the concept. On disclosure, the EU AI Act's transparency duty (Art. 13) and GPAI provider documentation obligations (Art. 53) raise the question of whether providers must publish hallucination rates; industry practice remains partial. On redress and accountability, Art. 85 and Art. 86 (right to explanation of individual decision-making), alongside OECD AI Principle 1.5 (accountability), frame the obligation, but operationalisation is inconsistent — and Novelli et al. 4 map exactly these liability, privacy, and IP gaps in how EU instruments apply to generative output. NIST's GenAI Profile (NIST AI 600-1) elevates 'confabulation' to a primary risk category, a deliberate vocabulary choice signalling resistance to anthropomorphic framing. Sectoral guidance dominates, with Brazil's PL 2338/2023 imposing accuracy obligations and UK AISI evaluations including factuality probes (Senado Federal 2023).
Sectoral Stakes: The Healthcare Test Case
Healthcare illustrates why hallucination drives the most concrete sectoral regulation: a confabulated dosage or fabricated citation presented as authoritative output can cause direct clinical harm. Weissman et al. 5 find general-purpose LLMs 'readily produced device-like decision support', implying they should fall under medical-device regulation if clinically deployed — a finding that collides with evidence 6 that most FDA AI/ML devices clear via the 510(k) pathway with limited clinical validation and poor transparency. Because hallucination evades pre-market testing, governance shifts toward post-market monitoring 7, while the WHO/ITU Global Initiative on AI for Health 8 pushes harmonised international standards across these fragmented national regimes.
Debates and Open Questions
Although the empirical consensus on the phenomenon is settled, its governance frontier is not. The liability surface has scaled from the single Mata v. Avianca (2023, S.D.N.Y.) hallucinated-citation matter to over 1,400 documented court cases with escalating sanctions, sharpening the unresolved allocation question among lawyer, provider, and deployer (Charlotin 2026). A second open question is whether mitigation — retrieval-augmented generation, confidence calibration, post-hoc verification — can ever satisfy a binding accuracy duty given that Xu et al. 3 hold elimination impossible. A third concerns redress design: contestability research 910 shows decision subjects need meaningful, channel-appropriate appeal, yet mandated human oversight is frequently a 'rubber-stamp' 11, leaving harms from confabulated output without reliable remedy.
Use in governance
How instruments operationalise this concept
| Instrument | Jurisdiction | Status |
|---|---|---|
| EU AI Act | EU | in force |
| NIST AI RMF Generative AI Profile | US | in force |
| Brazil AI Bill (PL 2338/2023) | BR | proposed |
| OECD AI Principles (Recommendation) | OECD | in force |
Appears in topic articles
Editorial note
NIST AI 600-1 prefers 'confabulation' over 'hallucination' to avoid anthropomorphic framing; the two terms are interchangeable in current technical literature but the policy-vocabulary choice signals editorial discipline. Wiki articles should default to 'hallucination' as the more widely-used term, but cite the NIST framing when paralleling AI 600-1. Currency (2026-06-21): Kalai/Nachum/Vempala/Zhang "Why Language Models Hallucinate" (arXiv:2509.04664, Sept 2025, w/ OpenAI blog) reframes the root cause as training/evaluation incentives that reward confident guessing over admitting uncertainty (fix: re-score leaderboards), and the legal-liability surface has scaled from the single Mata v. Avianca cite to 1,400+ documented court matters (~90% in 2025, escalating sanctions per Charlotin DB) — both worth adding; the definition itself remains accurate.
See also
Further reading
Sources on the broader topics this concept relates to — complementing, not standing in for, the primary sources cited inline above. 76 academic & grey-literature sources; catalogued metadata with a primary link; one-line findings are ✦ AI-generated summaries, labeled as such (charter §7.9). Browse the full literature index.
- Current state of Food and Drug Administration-approved artificial intelligence/machine learning medical devices: pathways, transparency, and evidence gaps Peer-reviewed✦ AIDocuments that most FDA AI/ML devices clear via the 510(k) pathway with limited clinical validation and poor transparency, exposing regulatory evidence gaps.
- An interdisciplinary account of the terminological choices by EU policymakers ahead of the final agreement on the AI Act: AI system, general purpose AI system, foundation model, and generative AI Peer-reviewed✦ AITraces how the AI Act's legal text shifted across versions among the terms 'AI system, general purpose AI system, foundation model, and generative AI', exposing definitional instability in the regime.
- The EU model of AI governance: regulating artificial intelligence through law and policy Peer-reviewed✦ AIAnalyses how the AI Act's risk-based model handles general-purpose and foundation models whose 'autonomous content generation challenges legal categories of authorship, accountability, and control'.
- Generative AI and data protection Peer-reviewed✦ AIExamines friction between foundation-model training and the GDPR, noting models that 'memorize and leak pieces of training data' cannot be treated as anonymous.
- Identifying Algorithmic Decision Subjects' Needs for Meaningful Contestability Peer-reviewed✦ AIEmpirically elicits what decision subjects need for contestation to be 'meaningful', informing the design of effective remedies and appeal mechanisms for ADM.
- Two Means to an End Goal: Connecting Explainability and Contestability in the Regulation of Public Sector AI Preprint✦ AIInterview study with 14 regulation experts distinguishes judicial vs non-judicial and individual vs collective contestation channels for public-sector AI remedies.
- Unregulated large language models produce medical device-like output Peer-reviewed✦ AIFinds general-purpose LLMs 'readily produced device-like decision support across a range of scenarios,' implying they should fall under medical-device regulation if clinically deployed.
- A general framework for governing marketed AI/ML medical devices Peer-reviewed✦ AIProposes a post-market governance framework for AI/ML medical devices addressing performance drift and ongoing monitoring beyond initial approval.
- Global Initiative on AI for Health (GI-AI4H): strategic priorities advancing governance across the United Nations Peer-reviewed✦ AISets out the WHO/ITU Global Initiative on AI for Health's strategic priorities to harmonize international regulatory and governance standards for health AI.
- GPTs are GPTs: Labor market impact potential of LLMs Peer-reviewed✦ AIFinds around 80% of the U.S. workforce "could have at least 10% of their work tasks affected" by LLMs, which exhibit "traits of general-purpose technologies".
- Generative AI in EU law: Liability, privacy, intellectual property, and cybersecurity Peer-reviewed✦ AIExamines how the EU AI Act, liability regimes, GDPR, copyright and cybersecurity rules apply to generative AI, identifying gaps and proposing targeted regulatory refinements.
- Evaluating Frontier Models for Dangerous Capabilities Preprint✦ AIPilots dangerous-capability evaluations (persuasion, cyber, self-proliferation) on frontier models, finding 'early warning signs' but no strong present danger — grounding evaluation-based gating.
+ 64 more across this concept's topics — see the literature index.
References
Sources cited inline in the analysis, numbered in order of appearance.
- Ji, Z., et al. (2023), 'Survey of Hallucination in Natural Language Generation,' ACM Computing Surveys 55(12): 1-38. Hallucination. arXiv:2202.03629 — Ji, Z., et al. (2023), 'Survey of Hallucination in Natural Language Generation,' ACM Computing Surveys 55(12): 1-38. ↩
- arXiv:2509.04664 ↩
- arXiv:2401.11817 ↩
- Novelli, Casolari, Hacker, Spedicato & Floridi (2024) Generative AI in EU law: Liability, privacy, intellectual property, and cybersecurity, Computer Law & Security Review. 10.1016/j.clsr.2024.106066 — Examines how the EU AI Act, liability regimes, GDPR, copyright and cybersecurity rules apply to generative AI, identifying gaps and proposing targeted regulatory refinements. ↩
- Gary E. Weissman, Toni Mankowitz, Genevieve P. Kanter (2025) Unregulated large language models produce medical device-like output, npj Digital Medicine. 10.1038/s41746-025-01544-y — Finds general-purpose LLMs 'readily produced device-like decision support across a range of scenarios,' implying they should fall under medical-device regulation if clinically deployed. ↩
- Aditya Loganathan, Michael Friedman, Tayab Waseem, et al. (Andrew C. Meltzer, senior author) (2026) Current state of Food and Drug Administration-approved artificial intelligence/machine learning medical devices: pathways, transparency, and evidence gaps, Journal of Medical Artificial Intelligence. 10.21037/jmai-2025-196 — Documents that most FDA AI/ML devices clear via the 510(k) pathway with limited clinical validation and poor transparency, exposing regulatory evidence gaps. ↩
- Boris Babic, I. Glenn Cohen, Ariel Dora Stern, Yiwen Li, Melissa Ouellet (2025) A general framework for governing marketed AI/ML medical devices, npj Digital Medicine. 10.1038/s41746-025-01717-9 — Proposes a post-market governance framework for AI/ML medical devices addressing performance drift and ongoing monitoring beyond initial approval. ↩
- Vijaytha Muralidharan, Madelena Y. Ng, Shada AlSalamah, Sameer Pujari, et al. (WHO/ITU GI-AI4H) (2025) Global Initiative on AI for Health (GI-AI4H): strategic priorities advancing governance across the United Nations, npj Digital Medicine. 10.1038/s41746-025-01618-x — Sets out the WHO/ITU Global Initiative on AI for Health's strategic priorities to harmonize international regulatory and governance standards for health AI. ↩
- Mireia Yurrita, Himanshu Verma, Agathe Balayn, Kars Alfrink, Ujwal Gadiraju, and Alessandro Bozzon (2025) Identifying Algorithmic Decision Subjects' Needs for Meaningful Contestability, Proceedings of the ACM on Human-Computer Interaction (CSCW). 10.1145/3757415 — Empirically elicits what decision subjects need for contestation to be 'meaningful', informing the design of effective remedies and appeal mechanisms for ADM. ↩
- arXiv:2504.18236 ↩
- Sarah Sterz, Kevin Baum, Sebastian Biewer, Holger Hermanns, Anne Lauber-Rönsberg, Philip Meinel, Markus Langer (2024) On the Quest for Effectiveness in Human Oversight: Interdisciplinary Perspectives, Proceedings of the 2024 ACM Conference on Fairness, Accounta. 10.1145/3630106.3659051 — Synthesises interdisciplinary evidence to argue that legally mandated human oversight of AI is often ineffective ('rubber-stamp') unless effectiveness conditions are explicitly designed for. ↩
Cite this article 8 formats · BibTeX, RIS, APA, Chicago, … · 1-click copy
Persistent identifier: https://policywindow.org/wiki/hallucination — committed-stable URL with content-versioning via ?asOf= (rollout pending per methodology §7). DOIs via Zenodo are on the roadmap.
Article tools — track changes, suggest an edit
View history — every captured revision of this article · What links here
Social-science evidence — the “so-what”
What the peer-reviewed social science shows: whether the harm this concept addresses is empirically real, and whether governance of it works. The badge is the epistemic status of the evidence(not the policy debate) — “thin” or “absent” efficacy evidence is itself a finding (the “second silence”). Each epistemic-status label is Policy Window's editorial assessment of the cited evidence base (a structured classification), not a verdict any single source issues.
LLM hallucination is empirically pervasive and quantitatively measured: TruthfulQA (Lin, Hilton & Evans 2022) shows models reproduce human falsehoods and misconceptions (best model truthful on 58% of questions vs 94% for humans), FActScore (Min et al. 2023) measures high rates of unsupported atomic facts in long-form generation on production models (e.g. ChatGPT biographies at ~58% factual precision), and Kalai & Vempala 2024 give a statistical lower-bound argument that calibrated language models must hallucinate ARBITRARY facts (those whose veracity cannot be determined from training data) at a nonzero rate. Caveat: measured rates vary widely by task, decoding strategy, and retrieval access, and the theoretical floor applies to arbitrary facts rather than all facts.
Sources: Lin, Hilton & Evans 2022 (TruthfulQA: Measuring How Models Mimic Human Falsehoods, ACL); Min et al. 2023 (FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation, EMNLP); Kalai & Vempala 2024 (Calibrated Language Models Must Hallucinate, STOC / arXiv:2311.14648)
TECHNICAL mitigations reduce but do not eliminate hallucination — retrieval augmentation lowers fabrication rates in knowledge-grounded dialogue (Shuster et al. 2021) — while Kalai & Vempala 2024 imply an irreducible floor for arbitrary facts under calibration. But these are engineering levers, not governance ones: there is no replicated impact evaluation showing that any disclosure mandate, labeling requirement, or regulatory regime measurably curbs downstream hallucination harm. So while the technical-mitigation evidence is solid, the evidence that any GOVERNANCE lever works is effectively absent; 'thin' here reflects that adjacent technical findings exist, not any demonstrated governance impact.
Sources: Shuster et al. 2021 (Retrieval Augmentation Reduces Hallucination in Conversation, EMNLP Findings); Kalai & Vempala 2024 (Calibrated Language Models Must Hallucinate, STOC / arXiv:2311.14648)