In-Context Learning
in-context-learning · Frontier safety
The capacity of a foundation model to adapt its behaviour to a new task purely from examples provided in the prompt, without any updates to the model's weights — discovered as an emergent property of large language models and now a primary evaluation surface.
Definition & scope
In-context learning (ICL) was named by Brown et al. (2020, 'Language Models are Few-Shot Learners,' the GPT-3 paper) as the surprising observation that sufficiently large language models could perform new tasks from a few demonstrations in the prompt. The phenomenon is empirically robust across scales above ~1B parameters; theoretical accounts (Xie et al. 2022, 'An Explanation of In-context Learning as Implicit Bayesian Inference'; Garg et al. 2022; von Oswald et al. 2023, 'Transformers Learn In-Context by Gradient Descent') propose various mechanisms but no consensus mechanism has emerged. Governance relevance is methodological. (a) Capability evaluations that test only baseline prompting under-state real-world capability, because deployment prompts routinely include task examples (Wei et al. 2022 chain-of-thought; Anil et al. 2024 many-shot). EU AI Act Art. 55(1)(a) adversarial testing must include ICL-mode probing to be capability-accurate. (b) Safety evaluations that test only baseline refusals under-state real-world failure surface, because many-shot jailbreaking exploits ICL to recover prohibited capabilities (Anil et al. 2024). (c) Model-card disclosures should specify which capabilities are baseline vs ICL-elicited (EU AIA Art. 53 transparency obligation). (d) ICL also affects the open-vs-closed debate: a closed model accessed via API still exposes ICL-elicitation surface, weakening the capability-containment assumption.
Precise Definition and Distinctions
In-context learning (ICL) denotes a foundation model's adaptation to a novel task purely from demonstrations placed in the prompt, with no gradient step and no change to stored weights. The term was coined in the GPT-3 paper 1 to label the surprising observation that few-shot prompting could match or rival task-specific tuning. The concept is best fixed by contrast with two adjacent adaptation routes. Fine-tuning alters the weight matrix and persists across sessions; retrieval-augmented generation injects retrieved passages rather than worked examples. All three raise deployed capability without retraining from scratch, but ICL is uniquely ephemeral and per-request — it leaves no auditable artifact in the model file, which is precisely why it complicates capability attestation and the definitional work that regulators are still doing 2.
Mechanisms and Empirical Substance
ICL is empirically robust above roughly 1B parameters but mechanistically unsettled. Competing accounts model it as implicit Bayesian inference over latent task variables (Xie et al. 2022), as the transformer approximating a learning algorithm over the demonstrations (Garg et al. 2022), and as forward passes that emulate gradient descent in activation space (von Oswald et al. 2023). No single mechanism commands consensus. The breadth of tasks ICL touches is consistent with treating large language models as general-purpose technologies that affect a wide swath of work 3. What is governance-salient is the dose-response behaviour: capability scales with the number and quality of in-prompt examples, extending to chain-of-thought prompting (Wei et al. 2022) and many-shot regimes with hundreds of demonstrations (Anil et al. 2024). Because the same surface that lifts benign accuracy can also resurrect suppressed behaviours, ICL is simultaneously a capability amplifier and an attack vector — an enhancement that lifts effective capability without proportional training compute 4, not a fixed model property a single baseline measurement can capture.
Governance Relevance and Engaged Provisions
ICL converts capability measurement from a property of the weights into a property of the prompt, which reshapes several obligations. Adversarial testing of general-purpose models under Art. 55(1)(a) of Regulation (EU) 2024/1689 is capability-accurate only if it probes ICL-elicited behaviour, since deployment prompts routinely carry examples; baseline-only probing understates the safety surface that many-shot jailbreaking exploits (Anil et al. 2024). Transparency duties under Art. 53 imply model cards should distinguish baseline from ICL-elicited capabilities. Frontier dangerous-capability evaluation 5 likewise gates on what elicitation can surface, not default prompting. ICL also bears on compute governance: techniques that lift effective capability without proportional training compute strain threshold-based regimes 46, since the prompt, not the FLOP count, moves the frontier.
Debates and Open Questions
Even with empirical consensus that ICL is real and scale-emergent, three disputes remain open. First, mechanism: the Bayesian, meta-learning, and gradient-descent readings make divergent predictions about robustness and failure, with no resolution (Xie et al. 2022; von Oswald et al. 2023). Second, open-versus-closed containment: API-gated access still exposes the ICL-elicitation surface, so weight secrecy is weaker containment than assumed — a closed model can be steered toward prohibited outputs by prompting alone. Third, governability: if effective capability moves with the prompt, static evaluation snapshots and document-bound transparency regimes age poorly; freedom-of-information regimes granting access only to existing documents struggle with adaptation that produces no document 7. Open problems in measurement and verification 8 bear on whether ICL-elicited capability can be reliably bounded.
Use in governance
How instruments operationalise this concept
| Instrument | Jurisdiction | Status |
|---|---|---|
| EU AI Act | EU | in force |
| NIST AI RMF Generative AI Profile | US | in force |
Appears in topic articles
Editorial note
Distinguish ICL (in-prompt example-based adaptation) from fine-tuning (weight-update-based adaptation) and from retrieval-augmented generation (retrieved-context-based adaptation). All three affect deployed capability without modifying the underlying model, but at different latencies + with different governance surfaces.
See also
Further reading
Sources on the broader topics this concept relates to — complementing, not standing in for, the primary sources cited inline above. 70 academic & grey-literature sources; catalogued metadata with a primary link; one-line findings are ✦ AI-generated summaries, labeled as such (charter §7.9). Browse the full literature index.
- An interdisciplinary account of the terminological choices by EU policymakers ahead of the final agreement on the AI Act: AI system, general purpose AI system, foundation model, and generative AI Peer-reviewed✦ AITraces how the AI Act's legal text shifted across versions among the terms 'AI system, general purpose AI system, foundation model, and generative AI', exposing definitional instability in the regime.
- The EU model of AI governance: regulating artificial intelligence through law and policy Peer-reviewed✦ AIAnalyses how the AI Act's risk-based model handles general-purpose and foundation models whose 'autonomous content generation challenges legal categories of authorship, accountability, and control'.
- Generative AI and data protection Peer-reviewed✦ AIExamines friction between foundation-model training and the GDPR, noting models that 'memorize and leak pieces of training data' cannot be treated as anonymous.
- Defending Compute Thresholds Against Legal Loopholes Preprint✦ AIIdentifies 'enhancement techniques that are capable of decreasing training compute usage while preserving... model capabilities', exposing loopholes in compute-reporting thresholds.
- GPTs are GPTs: Labor market impact potential of LLMs Peer-reviewed✦ AIFinds around 80% of the U.S. workforce "could have at least 10% of their work tasks affected" by LLMs, which exhibit "traits of general-purpose technologies".
- Computing Power and the Governance of Artificial Intelligence Preprint✦ AIArgues compute is a uniquely governable lever because it is "detectable, excludable, and quantifiable, and is produced via an extremely concentrated supply chain".
- Training Compute Thresholds: Features and Functions in AI Regulation Preprint✦ AIFinds "training compute currently is the most suitable metric to identify GPAI models", but thresholds should only trigger further scrutiny, not determine risk measures alone.
- Compute North vs. Compute South: The Uneven Possibilities of Compute-based AI Governance Around the Globe Peer-reviewed✦ AICensus of hyperscale cloud regions shows a divide between "Compute North" states hosting training-relevant compute and a Compute South, shaping who can wield compute-based governance.
- Generative AI in EU law: Liability, privacy, intellectual property, and cybersecurity Peer-reviewed✦ AIExamines how the EU AI Act, liability regimes, GDPR, copyright and cybersecurity rules apply to generative AI, identifying gaps and proposing targeted regulatory refinements.
- Evaluating Frontier Models for Dangerous Capabilities Preprint✦ AIPilots dangerous-capability evaluations (persuasion, cyber, self-proliferation) on frontier models, finding 'early warning signs' but no strong present danger — grounding evaluation-based gating.
- Governing Through the Cloud: The Intermediary Role of Compute Providers in AI Regulation Preprint✦ AIArgues 'compute providers should have legal obligations' to secure infrastructure, keep records, verify activity and report frontier training as regulatory intermediaries.
- Verification methods for international AI agreements Preprint✦ AISurveys '10 verification methods that could detect... unauthorized AI training... and unauthorized data centers', mapping the technical basis for compute-disclosure regimes.
+ 58 more across this concept's topics — see the literature index.
References
Sources cited inline in the analysis, numbered in order of appearance.
- Brown, T., et al. (2020), 'Language Models are Few-Shot Learners' (GPT-3 paper) — the canonical articulation of in-context learning as an emergent capability. In-Context Learning. arXiv:2005.14165 — Brown, T., et al. (2020), 'Language Models are Few-Shot Learners' (GPT-3 paper) — the canonical articulation of in-context learning as an emergent capability. ↩
- David Fernández-Llorca, Emilia Gómez, Ignacio Sánchez, Gabriele Mazzini (2025) An interdisciplinary account of the terminological choices by EU policymakers ahead of the final agreement on the AI Act: AI system, general purpose AI system, foundation model, and generative AI, Artificial Intelligence and Law. 10.1007/s10506-024-09412-y — Traces how the AI Act's legal text shifted across versions among the terms 'AI system, general purpose AI system, foundation model, and generative AI', exposing definitional instability in the regime. ↩
- Eloundou, Manning, Mishkin, Rock (2024) GPTs are GPTs: Labor market impact potential of LLMs, Science. 10.1126/science.adj0998 — Finds around 80% of the U.S. workforce "could have at least 10% of their work tasks affected" by LLMs, which exhibit "traits of general-purpose technologies". ↩
- Matteo Pistillo, Pablo Villalobos (2025) Defending Compute Thresholds Against Legal Loopholes, arXiv (cs.CY). arXiv:2502.00003 — Identifies 'enhancement techniques that are capable of decreasing training compute usage while preserving... model capabilities', exposing loopholes in compute-reporting thresholds. ↩
- Mary Phuong, Matthew Aitchison, Elliot Catt, et al. (Google DeepMind) (2024) Evaluating Frontier Models for Dangerous Capabilities, arXiv (cs.LG). arXiv:2403.13793 — Pilots dangerous-capability evaluations (persuasion, cyber, self-proliferation) on frontier models, finding 'early warning signs' but no strong present danger — grounding evaluation-based gating. ↩
- Heim & Koessler (2024) Training Compute Thresholds: Features and Functions in AI Regulation, arXiv. arXiv:2405.10799 — Finds "training compute currently is the most suitable metric to identify GPAI models", but thresholds should only trigger further scrutiny, not determine risk measures alone. ↩
- Henrik Palmer Olsen, Thomas Troels Hildebrandt, Cornelius Wiesener, Matthias Smed Larsen, Asbjørn William Ammitzbøll Flügge (2024) The Right to Transparency in Public Governance: Freedom of Information and the Use of Artificial Intelligence by Public Agencies, Digital Government: Research and Practice. 10.1145/3632753 — Finds freedom-of-information regimes "generally only grant access to existing documents" and that with "no mature standard for documenting AI models," public-sector AI transparency is limited. ↩
- Anka Reuel, Ben Bucknall, Stephen Casper, Tim Fist, Lennart Heim, et al. (34 authors) (2024) Open Problems in Technical AI Governance, arXiv (cs.CY). arXiv:2407.14981 — Catalogs open problems in 'technical analysis and tools for supporting the effective governance of AI', including compute measurement, verification and reporting gaps. ↩
Cite this article 8 formats · BibTeX, RIS, APA, Chicago, … · 1-click copy
Persistent identifier: https://policywindow.org/wiki/in-context-learning — committed-stable URL with content-versioning via ?asOf= (rollout pending per methodology §7). DOIs via Zenodo are on the roadmap.
Article tools — track changes, suggest an edit
View history — every captured revision of this article · What links here
Social-science evidence — the “so-what”
What the peer-reviewed social science shows: whether the harm this concept addresses is empirically real, and whether governance of it works. The badge is the epistemic status of the evidence(not the policy debate) — “thin” or “absent” efficacy evidence is itself a finding (the “second silence”). Each epistemic-status label is Policy Window's editorial assessment of the cited evidence base (a structured classification), not a verdict any single source issues.
In-context learning is robustly demonstrated at frontier scale: Brown et al. 2020 showed GPT-3 performing new tasks from prompt examples alone with no weight updates, and Olsson et al. 2022 give preliminary mechanistic evidence (their own framing) that induction heads may implement much of the phenomenon in large transformers. Caveat: Min et al. 2022 found demonstrations work largely by specifying the label space, input distribution, and format rather than by teaching the ground-truth input-label mapping (random labels barely hurt performance across 12 models including GPT-3), so ICL adapts to a task's surface form more than it 'learns' the mapping.
Sources: Brown et al. 2020 (Language Models are Few-Shot Learners, NeurIPS 2020 / arXiv:2005.14165); Olsson et al. 2022 (In-context Learning and Induction Heads, Transformer Circuits Thread, Anthropic); Min et al. 2022 (Rethinking the Role of Demonstrations, EMNLP 2022 / arXiv:2202.12837)
Because ICL is a capability rather than a harm, governance efficacy concerns controlling its misuse surface, where evidence that mitigations reliably work is thin. Anil et al. 2024 (Many-shot Jailbreaking) showed that scaling in-context demonstrations of harmful behavior elicits compliance at rates rising with example count, and reported that fine-tuning and prompt-based defenses reduced but did not eliminate the attack (fine-tuning only raises the number of demonstrations needed; prompt-based classification cut but did not zero out attack success), with larger context windows widening the surface. No replicated study shows a governance or technical regime durably bounds the safety-relevant uses of in-context learning.
Sources: Anil et al. 2024 (Many-shot Jailbreaking, NeurIPS 2024 / Anthropic)