Systemic Risk (AI)
systemic-risk · Risk classification
A regulatory designation indicating that a general-purpose AI model poses risks of significant scale or scope across the EU internal market, triggering Article 55 obligations under the EU AI Act.
Definition & scope
Article 51 of the EU AI Act establishes that a general-purpose AI (GPAI) model has systemic risk when its capabilities equal or exceed those of the most advanced models, evaluated via Annex XIII criteria. Presumption thresholds: ≥10²⁵ FLOPs training compute OR ≥45M EU monthly active users OR designation by the AI Office based on capability indicators. Designation triggers Article 55 obligations: model evaluation including adversarial testing, systemic risk assessment, incident reporting, cybersecurity protection, and energy reporting.
Locus of dispute: EU AIA's systemic-risk thresholds presume that capabilities ≥10²⁵ FLOPs OR ≥45M EU MAU correlate with systemic risk. Field is divided on whether either correlation is empirically validated; the catastrophic-risk literature uses a stricter definition (CBRN uplift, autonomous replication) that the EU AIA does not directly target.
Mechanism: what makes an AI risk "systemic"
The defining mechanism in the regulatory sense is propagation, not severity alone. Article 3(65) EU AIA pins "systemic risk" to harms "specific to the high-impact capabilities of general-purpose AI models" that "can be propagated at scale across the value chain" (Regulation (EU) 2024/1689, Art. 3(65)). The operative idea is that a single upstream model, adapted by thousands of downstream deployers, becomes a shared point of failure: a capability or flaw in the base model is inherited by every application built on it, so localized harm scales to society- or market-wide harm through the dependency graph. The trigger attaches to the model (Art. 51) rather than to any one use, operationalised through a training-compute proxy — a design choice defended on the ground that compute "currently is the most suitable metric to identify GPAI models" while only triggering further scrutiny rather than fixing risk by itself 1, and that compute is uniquely governable because it is "detectable, excludable, and quantifiable" 2.
The technical substance underneath this legal label is heterogeneous. Uuk et al. (2024), reviewing 86 papers, distil 13 distinct categories of GPAI systemic risk and 50 contributing sources, spanning loss of control, structural discrimination, governance failure, economic disruption, and environmental harm 3. Their analysis stresses that these risks arise less from a single malfunction than from knowledge gaps, difficulty in recognising diffuse harm, and the unpredictable trajectory of capability development. The practical implication, codified in the July 2025 GPAI Code of Practice, is that mitigation targets the upstream chokepoint — model evaluation, adversarial testing, and a pre-release Safety and Security Framework — on the theory that intervening at the propagation source contains downstream cascade more effectively than policing each deployment (EU GPAI Code of Practice, Safety & Security chapter, 10 July 2025) (European AI Office 2025).
History: a borrowed term, codified in stages
"Systemic risk" is not native to AI policy; it is a transplant with a datable path into EU digital law. The phrase originates in financial regulation, where it described contagion and cascading failure across interconnected institutions and gained prominence after successive banking crises (Financial Stability Board usage; the live article's editorial note). Its first migration into EU digital governance came not via the AI Act but the Digital Services Act: Article 34 DSA (Regulation (EU) 2022/2065) obliged very large online platforms to "identify, analyse and assess any systemic risks" from their services — notably without defining the term — across illegal content, fundamental rights, civic discourse, and public health/security.
The AI Act then borrowed the label and, unlike the DSA, supplied an explicit definition in Article 3(65) (Regulation (EU) 2024/1689, adopted 2024), tying it to high-impact GPAI capabilities and value-chain propagation, with the with-systemic-risk designation operationalised through Articles 51–55. That definition was itself the product of an unstable drafting process: the Act's text shifted across versions among "AI system, general purpose AI system, foundation model, and generative AI", a definitional instability traced by Fernández-Llorca et al. 4 and a recurring strain on the risk-based model's ability to handle general-purpose and foundation models 5. Implementation arrived in stages: the GPAI provisions became applicable in 2025, and the European AI Office published the final General-Purpose AI Code of Practice on 10 July 2025, whose Safety and Security chapter applies only to systemic-risk models — a group estimated at roughly 5–15 providers worldwide (EU GPAI Code of Practice, 10 July 2025) (European AI Office 2025). The compressed genealogy — finance → DSA (2022) → AIA (2024) → Code of Practice (2025) — is itself the basis for the recurring critique that the term carries financial-contagion connotations its AI usage does not actually establish.
Relation to adjacent concepts
"Systemic risk" is frequently conflated with three neighbours it is analytically distinct from. First, the financial sense: there it denotes contagion across interconnected institutions, where one failure cascades to others; the AIA reuses the word but its propagation channel is the model-to-deployer value chain, not inter-firm contagion — a correspondence the catalog flags as "theorized rather than empirically demonstrated" and that Hooker (2024) calls an uncertain compute-to-risk mapping 6, one further undercut by "enhancement techniques that are capable of decreasing training compute usage while preserving... model capabilities" and so opening loopholes in the compute proxy itself 7.
Second, structural risk in the AI-governance literature. Zwetsloot and Dafoe (2019) partition AI risk into misuse, accident, and structural — the last being harm that arises from how a technology reshapes incentives, power balances, and competitive dynamics even when no actor misuses it and nothing malfunctions ("Thinking About Risks From AI: Accidents, Misuse and Structure", GovAI). Systemic risk in the AIA sense overlaps with the structural category (both are diffuse and emergent) but is narrower: it is a legal designation gated on a specific model and compute proxy, whereas structural risk is a causal-pathway concept independent of any threshold.
Third, catastrophic / frontier risk. The catastrophic-risk literature defines its object by outcome severity — CBRN uplift, autonomous replication, loss of control — and empirical dangerous-capability evaluations of frontier models report "early warning signs" but no strong present danger 8. The AIA's systemic-risk trigger does not directly target those outcomes; it presumes risk from compute and reach (live article, locus of dispute). Thus a model can be "with systemic risk" under the AIA without meeting the catastrophic-risk literature's stricter criteria, and the two categories are not coextensive.
Use in governance
How instruments operationalise this concept
| Instrument | Jurisdiction | Status |
|---|---|---|
| EU AI Act | EU | in force |
| G7 Hiroshima AI Process Code of Conduct | G7 | in force |
| Council of Europe Framework Convention on AI | council_of_europe | adopted not in force |
Appears in topic articles
Editorial note
'Systemic risk' under the EU AIA is distinct from financial-system 'systemic risk' (SIFI/G-SIB regimes). Wiki articles in AI contexts default to the EU AIA usage. Currency (2026-06-21): The core EU AIA systemic-risk definition (Art. 3(65)/51/55, 10^25 FLOP rebuttable presumption + Annex XIII, contested compute proxy, July 2025 GPAI Code of Practice) is accurate and current, but the top-level scope/contestedQuestion fields wrongly list "≥45M EU monthly active users" as an AI Act Art. 51 trigger — that is the DSA VLOP threshold, not an AIA criterion (the article's own iter-443 evidenceBase already flags this self-contradiction); enforcement of GPAI systemic-risk obligations begins 2 Aug 2026 with ~12 models in scope via the compute presumption and still no public Art. 51(1)(b) qualitative designation.
See also
Further reading
Sources on the broader topics this concept relates to — complementing, not standing in for, the primary sources cited inline above. 68 academic & grey-literature sources; catalogued metadata with a primary link; one-line findings are ✦ AI-generated summaries, labeled as such (charter §7.9). Browse the full literature index.
- An interdisciplinary account of the terminological choices by EU policymakers ahead of the final agreement on the AI Act: AI system, general purpose AI system, foundation model, and generative AI Peer-reviewed✦ AITraces how the AI Act's legal text shifted across versions among the terms 'AI system, general purpose AI system, foundation model, and generative AI', exposing definitional instability in the regime.
- The EU model of AI governance: regulating artificial intelligence through law and policy Peer-reviewed✦ AIAnalyses how the AI Act's risk-based model handles general-purpose and foundation models whose 'autonomous content generation challenges legal categories of authorship, accountability, and control'.
- Generative AI and data protection Peer-reviewed✦ AIExamines friction between foundation-model training and the GDPR, noting models that 'memorize and leak pieces of training data' cannot be treated as anonymous.
- Defending Compute Thresholds Against Legal Loopholes Preprint✦ AIIdentifies 'enhancement techniques that are capable of decreasing training compute usage while preserving... model capabilities', exposing loopholes in compute-reporting thresholds.
- Identifying Algorithmic Decision Subjects' Needs for Meaningful Contestability Peer-reviewed✦ AIEmpirically elicits what decision subjects need for contestation to be 'meaningful', informing the design of effective remedies and appeal mechanisms for ADM.
- Two Means to an End Goal: Connecting Explainability and Contestability in the Regulation of Public Sector AI Preprint✦ AIInterview study with 14 regulation experts distinguishes judicial vs non-judicial and individual vs collective contestation channels for public-sector AI remedies.
- GPTs are GPTs: Labor market impact potential of LLMs Peer-reviewed✦ AIFinds around 80% of the U.S. workforce "could have at least 10% of their work tasks affected" by LLMs, which exhibit "traits of general-purpose technologies".
- Computing Power and the Governance of Artificial Intelligence Preprint✦ AIArgues compute is a uniquely governable lever because it is "detectable, excludable, and quantifiable, and is produced via an extremely concentrated supply chain".
- Training Compute Thresholds: Features and Functions in AI Regulation Preprint✦ AIFinds "training compute currently is the most suitable metric to identify GPAI models", but thresholds should only trigger further scrutiny, not determine risk measures alone.
- Compute North vs. Compute South: The Uneven Possibilities of Compute-based AI Governance Around the Globe Peer-reviewed✦ AICensus of hyperscale cloud regions shows a divide between "Compute North" states hosting training-relevant compute and a Compute South, shaping who can wield compute-based governance.
- Generative AI in EU law: Liability, privacy, intellectual property, and cybersecurity Peer-reviewed✦ AIExamines how the EU AI Act, liability regimes, GDPR, copyright and cybersecurity rules apply to generative AI, identifying gaps and proposing targeted regulatory refinements.
- Evaluating Frontier Models for Dangerous Capabilities Preprint✦ AIPilots dangerous-capability evaluations (persuasion, cyber, self-proliferation) on frontier models, finding 'early warning signs' but no strong present danger — grounding evaluation-based gating.
+ 56 more across this concept's topics — see the literature index.
References
Sources cited inline in the analysis (linked from the superscript markers), then the primary instrument sources behind the classifications.
- Heim & Koessler (2024) Training Compute Thresholds: Features and Functions in AI Regulation, arXiv. arXiv:2405.10799 — Finds "training compute currently is the most suitable metric to identify GPAI models", but thresholds should only trigger further scrutiny, not determine risk measures alone. ↩
- Sastry, Heim, Belfield, Anderljung, Brundage, et al. (2024) Computing Power and the Governance of Artificial Intelligence, arXiv. arXiv:2402.08797 — Argues compute is a uniquely governable lever because it is "detectable, excludable, and quantifiable, and is produced via an extremely concentrated supply chain". ↩
- arXiv:2412.07780 ↩
- David Fernández-Llorca, Emilia Gómez, Ignacio Sánchez, Gabriele Mazzini (2025) An interdisciplinary account of the terminological choices by EU policymakers ahead of the final agreement on the AI Act: AI system, general purpose AI system, foundation model, and generative AI, Artificial Intelligence and Law. 10.1007/s10506-024-09412-y — Traces how the AI Act's legal text shifted across versions among the terms 'AI system, general purpose AI system, foundation model, and generative AI', exposing definitional instability in the regime. ↩
- Martina Hulok (2025) The EU model of AI governance: regulating artificial intelligence through law and policy, ERA Forum. 10.1007/s12027-025-00869-1 — Analyses how the AI Act's risk-based model handles general-purpose and foundation models whose 'autonomous content generation challenges legal categories of authorship, accountability, and control'. ↩
- arXiv:2407.05694 ↩
- Matteo Pistillo, Pablo Villalobos (2025) Defending Compute Thresholds Against Legal Loopholes, arXiv (cs.CY). arXiv:2502.00003 — Identifies 'enhancement techniques that are capable of decreasing training compute usage while preserving... model capabilities', exposing loopholes in compute-reporting thresholds. ↩
- Mary Phuong, Matthew Aitchison, Elliot Catt, et al. (Google DeepMind) (2024) Evaluating Frontier Models for Dangerous Capabilities, arXiv (cs.LG). arXiv:2403.13793 — Pilots dangerous-capability evaluations (persuasion, cyber, self-proliferation) on frontier models, finding 'early warning signs' but no strong present danger — grounding evaluation-based gating. ↩
- Regulation (EU) 2024/1689, Arts. 51-55
Cite this article 8 formats · BibTeX, RIS, APA, Chicago, … · 1-click copy
Persistent identifier: https://policywindow.org/wiki/systemic-risk — committed-stable URL with content-versioning via ?asOf= (rollout pending per methodology §7). DOIs via Zenodo are on the roadmap.
Article tools — track changes, suggest an edit
View history — every captured revision of this article · What links here
Social-science evidence — the “so-what”
What the peer-reviewed social science shows: whether the harm this concept addresses is empirically real, and whether governance of it works. The badge is the epistemic status of the evidence(not the policy debate) — “thin” or “absent” efficacy evidence is itself a finding (the “second silence”). Each epistemic-status label is Policy Window's editorial assessment of the cited evidence base (a structured classification), not a verdict any single source issues.
The EU AIA codifies 'systemic risk' for GPAI (Art. 3(65) — 'a risk specific to the high-impact capabilities of GPAI models ... that can be propagated at scale across the value chain') and triggers the with-systemic-risk designation via a rebuttable presumption at >10^25 training FLOP plus Annex XIII reach/impact criteria and Commission discretion (Art. 51). The category names something real — the most capable models do cluster above this compute frontier — but its central proxy is contested: Hooker 2024 argues training-FLOP thresholds are shortsighted and rest on a 'highly uncertain' compute-to-risk relationship (an uncertain, arguably gameable proxy for capability or risk), and the AIA borrows the term 'systemic risk' from finance (FSB: disruption/cascading-failure/contagion across an interconnected financial system) where it carries a distinct contagion meaning, so the designation's correspondence to actual systemic harm is theorized rather than empirically demonstrated. Caveat: the commonly-cited '45M MAU' criterion is the DSA VLOP designation threshold, not an AIA one — the AIA's reach proxy is Annex XIII's >=10,000 registered EU business users.
Sources: EU AI Act Art. 3(65), Art. 51, Annex XIII (artificialintelligenceact.eu); Hooker 2024 (On the Limitations of Compute Thresholds as a Governance Strategy, arXiv:2407.05694)
There is no impact evaluation showing that the systemic-risk designation or its obligations (the Art. 55 duties operationalized by the July 2025 GPAI Code of Practice — Safety & Security Framework, model/adversarial evaluation, serious incident reporting) actually reduce systemic harm; the regime only came into application in 2025-2026 and its core trigger is argued to be a poor risk proxy (Hooker 2024). Proposed mitigations are surveyed only as expert-perceived plausibility (Uuk et al. 2024 — an expert-judgment study, not an outcome study), so the evidence that this governance lever actually works is effectively absent.
Sources: Hooker 2024 (Limitations of Compute Thresholds, arXiv:2407.05694); EU GPAI Code of Practice, Safety & Security chapter, July 2025 (artificialintelligenceact.eu / code-of-practice.ai); Uuk et al. 2024 (Effective Mitigations for Systemic Risks from General-Purpose AI, arXiv:2412.02145)