Catastrophic & Existential Risk is governance of model capabilities that could cause mass casualties or civilisational-scale harms (CBRN uplift, autonomous replication, deceptive alignment). Distinct from EU AIA 'systemic risk' which targets market-scale rather than catastrophic-scale harms. Across 26 tracked AI-governance instruments, 11 address this topic explicitly, 6 via general principles, and 9 are silent.
Definition and scope
Governance of model capabilities that could cause mass casualties or civilisational-scale harms (CBRN uplift, autonomous replication, deceptive alignment). Distinct from EU AIA 'systemic risk' which targets market-scale rather than catastrophic-scale harms.
The cross-jurisdiction picture below shows how each of 26 tracked instruments treats this topic. The patterns vary substantially — and 9 regimes are silent, leaving gaps that future policy work could address.
Historical primacy & cross-jurisdiction tension
First addressed by NIST AI Risk Management Framework on (implicit). Subsequent regimes have either codified, diverged from, or remained silent on this baseline.
- Forum-shoppingExecutive Order 14110 on Safe, Secure, Trustworthy AI↔Executive Order 14179 — Removing Barriers to American Leadership in AI
- Forum-shoppingG7 Hiroshima AI Process Code of Conduct↔Interim Measures for Generative AI Service Management
- Forum-shoppingBletchley Declaration on AI Safety↔OECD AI Principles (Recommendation)
Cross-jurisdiction coverage
| Instrument | Jurisdiction | Coverage | Citation |
|---|---|---|---|
| EU AI Act | EU | implicit | Art. 51 + Recital 32 — systemic risk overlaps with but does not fully cover catastrophic-risk framing |
| Executive Order 14110 on Safe, Secure, Trustworthy AI | US | governs | §4.2(a)(ii) — CBRN + autonomous replication explicitly named |
| Executive Order 14179 — Removing Barriers to American Leadership in AI | US | silent | — |
| UK Pro-Innovation Approach to AI Regulation (White Paper) | UK | implicit | AISI remit covers frontier-model evaluation; not in white paper text |
| Interim Measures for Generative AI Service Management | CN | silent | PRC framing uses 'safety + security' broadly, not catastrophic risk |
| G7 Hiroshima AI Process Code of Conduct | G7 | governs | Code §1 + §3 — explicit risk-identification including CBRN |
| OECD AI Principles (Recommendation) | OECD | silent | 2019 vintage — predates the 2023+ catastrophic-risk policy turn |
| Council of Europe Framework Convention on AI | council_of_europe | silent | Treaty focuses on individual rights, not catastrophic-system risks |
| UN GA Resolution on Safe, Secure, Trustworthy AI | UN | implicit | Notes 'shared concerns' but no operative catastrophic-risk text |
| NIST AI Risk Management Framework | US | implicit | Map 1.1 risk classification covers catastrophic via 'societal' impact tier; GenAI Profile (2024) adds explicit content |
| Bletchley Declaration on AI Safety | global | governs | Declaration §3-5 (substantial risks from frontier AI, including catastrophic harm) |
| Seoul Declaration on Safe, Innovative and Inclusive AI | global | governs | Frontier AI Safety Commitments §1: identify thresholds for severe risks pre-deployment |
| NIST AI RMF Generative AI Profile | US | governs | NIST AI 600-1 §3.1 CBRN Information Uplift; §3.3 Dangerous, Violent, or Hateful Content |
| California SB-1047: Safe and Secure Innovation for Frontier AI Models Act | US | governs | Cal. SB-1047 §22602 — defines 'critical harm' including mass casualties, $500M+ damage |
| India Digital Personal Data Protection Act + AI Advisory (MEITY) | IN | silent | — |
| Brazil AI Bill (PL 2338/2023) | BR | governs | PL 2338/2023 Art. 14 (excessive-risk AI applications — explicit prohibition + risk-tier framework) |
| ASEAN Guide on AI Governance and Ethics | ASEAN | silent | — |
| African Union Continental AI Strategy | African_Union | silent | — |
| Anthropic Responsible Scaling Policy (RSP) v2 | US | governs | RSP v2 §3 — ASL-3 / ASL-4 capability thresholds explicitly target CBRN uplift + autonomous-replication |
| OpenAI Preparedness Framework | US | governs | Preparedness Framework risk-tier matrix — Critical tier explicitly targets CBRN, cyber, persuasion, autonomy |
| Google DeepMind Frontier Safety Framework | US | governs | FSF Critical Capability Levels (CCL) — explicit thresholds for autonomy, biosecurity, cyber, persuasion |
| Meta Frontier AI Framework | US | governs | Framework critical-risk tier — commit to halt training pre-mitigation if reached |
| UK-US AI Safety Institute Memorandum of Understanding | global | implicit | Joint evaluation scope encompasses CBRN + autonomy uplift questions; MoU text does not enumerate explicit thresholds |
| White House Voluntary AI Commitments | US | implicit | Commitments §1 references CBRN + bio risks via 'most significant societal risks'; not threshold-explicit |
| Singapore Model AI Governance Framework for Generative AI | SG | silent | — |
| Japan METI AI Guidelines for Business | JP | silent | — |
Silent regimes — gap signal
Instruments that do not address Catastrophic & Existential Risk — candidates for future policy work.
- Executive Order 14179 — Removing Barriers to American Leadership in AIUS
- Interim Measures for Generative AI Service ManagementCN
- OECD AI Principles (Recommendation)OECD
- Council of Europe Framework Convention on AIcouncil_of_europe
- India Digital Personal Data Protection Act + AI Advisory (MEITY)IN
- ASEAN Guide on AI Governance and EthicsASEAN
- African Union Continental AI StrategyAfrican_Union
- Singapore Model AI Governance Framework for Generative AISG
- Japan METI AI Guidelines for BusinessJP
Related topics
Topics with similar coverage patterns across the tracked instruments.
References
- EU-AIA-2024: Art. 51 + Recital 32 — systemic risk overlaps with but does not fully cover catastrophic-risk framing
- US-EO-14110: §4.2(a)(ii) — CBRN + autonomous replication explicitly named
- UK-WHITEPAPER-2023: AISI remit covers frontier-model evaluation; not in white paper text
- G7-HIROSHIMA: Code §1 + §3 — explicit risk-identification including CBRN
- UN-RES-2024: Notes 'shared concerns' but no operative catastrophic-risk text
- NIST-AI-RMF: Map 1.1 risk classification covers catastrophic via 'societal' impact tier; GenAI Profile (2024) adds explicit content
- BLETCHLEY-2023: Declaration §3-5 (substantial risks from frontier AI, including catastrophic harm)
- SEOUL-2024: Frontier AI Safety Commitments §1: identify thresholds for severe risks pre-deployment
- NIST-AI-RMF-GENAI: NIST AI 600-1 §3.1 CBRN Information Uplift; §3.3 Dangerous, Violent, or Hateful Content
- CA-SB-1047: Cal. SB-1047 §22602 — defines 'critical harm' including mass casualties, $500M+ damage
- BR-AIBILL-2024: PL 2338/2023 Art. 14 (excessive-risk AI applications — explicit prohibition + risk-tier framework)
- ANTHROPIC-RSP-2024: RSP v2 §3 — ASL-3 / ASL-4 capability thresholds explicitly target CBRN uplift + autonomous-replication
- OPENAI-PREPAREDNESS-2023: Preparedness Framework risk-tier matrix — Critical tier explicitly targets CBRN, cyber, persuasion, autonomy
- DEEPMIND-FSF-2024: FSF Critical Capability Levels (CCL) — explicit thresholds for autonomy, biosecurity, cyber, persuasion
- META-FRONTIER-2024: Framework critical-risk tier — commit to halt training pre-mitigation if reached
- UK-US-AISI-MOU-2024: Joint evaluation scope encompasses CBRN + autonomy uplift questions; MoU text does not enumerate explicit thresholds
- WH-VOLUNTARY-2023: Commitments §1 references CBRN + bio risks via 'most significant societal risks'; not threshold-explicit
Take this further — sign up free
Save, compare, or get alerts when Catastrophic & Existential Risk changes. Policy Window is the analyst workbench layered on top of this wiki — free for researchers, civil society, and verified policymakers.
17 instruments tracked.