Cross-corpus research synthesis
Catastrophic & Existential Risk
Governance of model capabilities that could cause mass casualties or civilisational-scale harms (CBRN uplift, autonomous replication, deceptive alignment). Distinct from EU AIA 'systemic risk' which targets market-scale rather than catastrophic-scale harms.
Synthesised deterministically from 27 articles that engage this theme. Empirical consensus: contested · contested: Are current frontier-model capabilities a meaningful contribution to catastrophic-risk probability? Field is split between catastrophic-risk-as-imminent (FLI, CAIS) and catastrophic-risk-as-speculative (Pope et al., Andersson) positions.. Full theme article: /wiki/catastrophic-risk. Machine-readable: /wiki/synthesis.json.
Cross-jurisdiction stances (14 govern, 22 engage)
| Instrument | Verdict | Provision excerpt / citation |
|---|---|---|
| EU AI Act | implicit | 'systemic risk' means a risk specific to the high-impact capabilities of general-purpose AI models … with significant impact on the Union market or on public health, safety, security, or fundamental rights… (paraphrase) Art. 51 + Recital 32 — systemic risk overlaps with but does not fully cover catastrophic-risk framing |
| Executive Order 14110 on Safe, Secure, Trustworthy AI | governs | §4.2(a)(ii) — CBRN + autonomous replication explicitly named |
| UK Pro-Innovation Approach to AI Regulation (White Paper) | implicit | AISI remit covers frontier-model evaluation; not in white paper text |
| G7 Hiroshima AI Process Code of Conduct | governs | Code §1 + §3 — explicit risk-identification including CBRN |
| UN GA Resolution on Safe, Secure, Trustworthy AI | implicit | Notes 'shared concerns' but no operative catastrophic-risk text |
| NIST AI Risk Management Framework | implicit | “Processes, procedures, and practices are in place to determine the needed level of risk management activities based on the organization's risk tolerance.” Map 1.1 risk classification covers catastrophic via 'societal' impact tier; GenAI Profile (2024) adds explicit content |
| Bletchley Declaration on AI Safety | governs | Declaration §3-5 (substantial risks from frontier AI, including catastrophic harm) |
| Seoul Declaration on Safe, Innovative and Inclusive AI | governs | Frontier AI Safety Commitments §1: identify thresholds for severe risks pre-deployment |
| NIST AI RMF Generative AI Profile | governs | NIST AI 600-1 §3.1 CBRN Information Uplift; §3.3 Dangerous, Violent, or Hateful Content |
| California SB-1047: Safe and Secure Innovation for Frontier AI Models Act | governs | Cal. SB-1047 §22602 — defines 'critical harm' including mass casualties, $500M+ damage |
| Brazil AI Bill (PL 2338/2023) | governs | PL 2338/2023 Art. 14 (excessive-risk AI applications — explicit prohibition + risk-tier framework) |
| Anthropic Responsible Scaling Policy (RSP) v2 | governs | RSP v2 §3 — ASL-3 / ASL-4 capability thresholds explicitly target CBRN uplift + autonomous-replication |
| OpenAI Preparedness Framework | governs | Preparedness Framework risk-tier matrix — Critical tier explicitly targets CBRN, cyber, persuasion, autonomy |
| Google DeepMind Frontier Safety Framework | governs | FSF Critical Capability Levels (CCL) — explicit thresholds for autonomy, biosecurity, cyber, persuasion |
| Meta Frontier AI Framework | governs | Framework critical-risk tier — commit to halt training pre-mitigation if reached |
| UK-US AI Safety Institute Memorandum of Understanding | implicit | Joint evaluation scope encompasses CBRN + autonomy uplift questions; MoU text does not enumerate explicit thresholds |
| White House Voluntary AI Commitments | implicit | Commitments §1 references CBRN + bio risks via 'most significant societal risks'; not threshold-explicit |
| EU General-Purpose AI Code of Practice | governs | Chapter 3 systemic-risk-tier capability evaluations + serious-incident reporting + model-weight access controls (Art. 55 substrate) |
| DoD Responsible AI Strategy and Implementation Pathway | implicit | “The Department's AI capabilities will have explicit, well-defined uses, and the safety, security, and effectiveness of such capabilities will be subject to testing and assurance within those defined uses…” Ethical Principle 'Reliable' + Tenet 4 (Requirements Validation) — JCIDS gating addresses mission-risk; DoDD 3000.09 separately governs autonomy-in-weapons LAWS-specific catastrophic-risk decisions |
| California SB-53: Transparency in Frontier Artificial Intelligence Act (TFAIA) | governs | 'Catastrophic risk' means a foreseeable and material risk that a frontier developer's … frontier model will materially contribute to the death of, or serious injury to, more than 50 people or more than one billion dollars ($1,000,000,000) in damage to, or loss of, property… (paraphrase) Bus. & Prof. Code § 22757.11 (definition) operationalized by §§ 22757.12 (framework) + 22757.13 (critical-safety-incident reporting to CalOES) |
| New York RAISE Act: Responsible AI Safety and Education Act | governs | A large developer shall implement a written safety and security protocol [addressing the risk of critical harm] and conspicuously publish it with appropriate redactions, transmitting a copy to the attorney general. (paraphrase) N.Y. Gen. Bus. Law § 1421(1) requires a large developer to implement and conspicuously publish a written safety and security protocol governing the risk of 'critical harm' from its frontier models, and § 1421(4) requires disclosure of safety incidents within 72 hours; § 1420(7) defines critical harm (100+ deaths/serious injuries or $1B damage via CBRN weapons or autonomous model conduct). NOTE: the floor-text § 1421(2) deployment PROHIBITION was struck by the chapter amendment enacted Mar. 27, 2026 (S8828/A9449), which reoriented the Act to a transparency-and-reporting regime; this cell tracks the RETAINED safety-protocol + incident-reporting duties, not a deployment ban. |
| UN Global Digital Compact | implicit | Assess the future directions and implications of artificial intelligence systems ... evidence-based impact, risk and opportunity assessments. (paraphrase) GDC Objective 5, paras 55(a) and 56(a) (A/RES/79/1, Annex I) |
Evidence convergence
Sources the corpus cites for this theme across multiple articles — a scientometric consensus signal computed from inline prose citations (the more articles independently cite a source, the more load-bearing it is for this theme). 33 sources are cited by ≥2 articles.
- 20×An interdisciplinary account of the terminological choices by EU policymakers ahead of the final agreement on the AI Act: AI system, general purpose AI system, foundation model, and generative AI — cited by 20 articles
- 18×The EU model of AI governance: regulating artificial intelligence through law and policy — cited by 18 articles
- 17×Artificial intelligence and synthetic biology: biosecurity risks, dual-use concerns, and governance pathways — cited by 17 articles
- 15×Two types of AI existential risk: decisive and accumulative — cited by 15 articles
- 12×Governing AI Agents — cited by 12 articles
- 11×Multi-Agent Risks from Advanced AI — cited by 11 articles
- 10×Infrastructure for AI Agents — cited by 10 articles
- 10×Missing the Mark: Adoption of Watermarking for Generative AI Systems in Practice and Implications Under the New EU AI Act — cited by 10 articles
- 10×Defending Compute Thresholds Against Legal Loopholes — cited by 10 articles
- 10×International Agreements on AI Safety: Review and Recommendations for a Conditional AI Safety Treaty — cited by 10 articles
- 9×Generative AI and data protection — cited by 9 articles
- 5×Audio deepfakes and the regulation of the landlords of creativity — cited by 5 articles
- 5×Managing extreme AI risks amid rapid progress — cited by 5 articles
- 5×AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agents — cited by 5 articles
- 5×Identifying Algorithmic Decision Subjects' Needs for Meaningful Contestability — cited by 5 articles
- 4×arxiv:2504.18236 — cited by 4 articles
- 4×Open Foundation Models and TDM Exceptions to Copyright – Building Blocks for an AI Ecosystem — cited by 4 articles
- 4×Artificial Intelligence and Nuclear Weapons Proliferation: The Technological Arms Race for (In)visibility — cited by 4 articles
- 4×A Teleological Interpretation of the Definition of DeepFakes in the EU Artificial Intelligence Act—A Purpose-Based Approach to Potential Problems With the Word 'Existing' — cited by 4 articles
- 4×The Current Landscape of Deepfake Legislation in the United States — cited by 4 articles