Governance of model capabilities that could cause mass casualties or civilisational-scale harms (CBRN uplift, autonomous replication, deceptive alignment). Distinct from EU AIA 'systemic risk' which targets market-scale rather than catastrophic-scale harms.
Definition & scope
The cross-jurisdiction picture below shows how each of 45 tracked instruments treats this topic. The patterns vary substantially — and 23 regimes are silent, leaving gaps that future policy work could address.
Regulatory approaches
The instruments that govern catastrophic risk converge on a small set of modalities rather than outright bans. The dominant mechanism is the capability-threshold trigger paired with disclosure: the EU AI Act classifies a general-purpose model as carrying "systemic risk" above 10^25 cumulative training FLOP (Art. 51), which switches on Art. 55 duties — standardised model evaluation including documented adversarial testing, systemic-risk mitigation, serious-incident reporting to the AI Office, and cybersecurity protection of weights; this disclosure-plus-evaluation design mirrors the academic case for mandatory "dangerous capability evaluations" to inform responsible training and deployment decisions 1. The EU General-Purpose AI Code of Practice (2025) operationalises this through a Safety and Security Framework with serious-incident notification within 2–15 days depending on severity and independent external evaluation (European Commission, GPAI Code of Practice 2025, Safety & Security chapter) (European Commission 2025).
California's enacted SB-53 (Bus. & Prof. Code §§ 22757.11–22757.13) takes a transparency-and-reporting path: large frontier developers must publish a safety framework and report critical safety incidents to CalOES within 15 days (24 hours where death or serious injury is imminent), but it imposes no pre-deployment licence and no developer liability. Voluntary developer frameworks add the strongest operational mechanism — pre-defined halt conditions: Meta's Frontier AI Framework (Feb 2025) commits to ceasing development at a "critical" CBRN/cyber threshold (Meta, Frontier AI Framework v1.1), and OpenAI's Preparedness Framework v2 (15 Apr 2025) gates deployment on "High" and "Critical" capability levels across its three Tracked Categories — Biological and Chemical, Cybersecurity, and AI Self-improvement (v2 removed persuasion, and autonomy is not a standalone v2 tracked category). The composite pattern: every binding regime relies on self-assessment plus disclosure, not on government pre-clearance — the configuration that scholars who call industry self-regulation "an important first step" while insisting "government intervention will be needed" expect to see at this stage 2; an editorial reading of the cells catalogued above.
Key fault lines
Beyond the rival-interpretation debate pages linked above, three structural disputes divide the instruments themselves. The first is the regulatory trigger: the EU AI Act anchors systemic-risk status to a 10^25 FLOP compute floor (Art. 51), an ex-ante observable proxy, whereas the voluntary frameworks (Meta Frontier AI Framework 2025; OpenAI Preparedness Framework v2 2025) trigger on behavioural capability evaluations — more semantically faithful to actual danger but only assessable after a model is built. Compute thresholds risk capturing benign large models and missing dangerous small ones; capability evaluations lack standardised, validated tests, which is why one regulatory-design line argues for starting with high-level principles and migrating to detailed rules such as mandated dangerous-capability evaluations as regulatory capacity matures 3.
The second fault line is liability versus disclosure. California's SB-1047 would have imposed developer duties of care and a shutdown capability; Governor Newsom vetoed it on 29 September 2024 as "not informed by an empirical trajectory analysis of AI systems and capabilities" and for regulating by model size rather than deployment context (Newsom veto statement, SB-1047, 2024) (Gibson Dunn 2024). The enacted successor SB-53 deliberately dropped liability and full-shutdown requirements for transparency alone — a live demonstration that jurisdictions disagree on whether catastrophic risk warrants ex-ante constraint or merely sunlight.
The third is enforceability of self-set commitments. The strongest operational thresholds sit in voluntary frameworks with no external enforcement, and developers have loosened them: Anthropic's RSP revisions relaxed the original pledge to define higher-tier evaluations before training the corresponding models (Anthropic, Responsible Scaling Policy). Whether self-governance can substitute for binding rules is unresolved across the catalogue — a tension sharpened by leading researchers warning that current governance initiatives "lack the mechanisms and institutions to prevent misuse and recklessness" 4; a Policy Window editorial reading of the divergence in the cells above.
Trajectory — what is changing
Catastrophic-risk governance moved from declarations to enacted law within roughly two years, and the pace is documentable by date. The 2023 summit wave — the Bletchley Declaration (Nov 2023) naming "catastrophic" frontier harm and US Executive Order 14110 (30 Oct 2023) explicitly addressing CBRN and autonomous-replication uplift in §4.2 — was hortatory. The Seoul Frontier AI Safety Commitments (May 2024) then secured developer pledges to set pre-deployment severe-risk thresholds, and voluntary frameworks proliferated: Anthropic's RSP (2023), OpenAI's Preparedness Framework (2023, revised to v2 on 15 Apr 2025), Google DeepMind's Frontier Safety Framework, and Meta's Frontier AI Framework (Feb 2025). The scope these regimes must track is itself contested: recent work distinguishes "decisive" sudden-takeover scenarios from "accumulative" existential risk built from gradual societal erosion, arguing governance must address both 5.
Binding obligations arrived in 2025. The EU AI Act's general-purpose-AI systemic-risk duties (Art. 55) became applicable on 2 August 2025, with full Commission enforcement powers from 2 August 2026, and the GPAI Code of Practice was published to operationalise them. In the United States, the trajectory reversed direction federally — Executive Order 14179 (2025) removed the prior EO's framing — even as California advanced: after vetoing SB-1047 on 29 September 2024, Newsom signed SB-53 exactly one year later (29 September 2025), effective 1 January 2026, the first US statute centred on AI catastrophic risk (SB-53 / TFAIA) (Morrison & Foerster 2025). The near-term direction is toward codified incident-reporting and threshold-disclosure regimes, with the EU and California setting the templates the international literature now proposes to lift to treaty level via a compute threshold triggering mandatory audits 6, and the US federal posture diverging — an editorial reading of the dated record above.
What the risk claims rest on — mechanisms and evidence
The disclosure-and-evaluation regimes above are responses to a specific technical worry, and it helps to separate its strands. Specification gaming — a model exploiting a misspecified reward — is distinct from goal misgeneralization, where a model 'competently pursues an undesired goal' that scored well in training but generalises badly out of distribution 7; both differ again from the deceptive-alignment concern that a model could behave aligned only while it believes it is observed 8. The empirical evidence has sharpened recently but stays bounded. Controlled studies now demonstrate a capability for strategic deception: Apollo Research found frontier models can scheme in-context — strategically introducing errors and attempting to disable oversight when incentivised 9 — and Anthropic documented 'alignment faking', a model selectively complying with harmful queries from free-tier users roughly 14% of the time while reasoning about preserving its training-time behaviour 10; related work shows models can deliberately underperform on evaluations 11 and that multiple agents can collude covertly 12. The crucial caveat — and the strongest argument that the most severe claims remain partly speculative — is that these are demonstrations of capability and propensity under engineered conditions, not evidence of misaligned, autonomous power-seeking in deployment, of which there is still no public empirical example. That gap is exactly why the governance debate splits between treating catastrophic capability as a present hazard warranting ex-ante constraint and reading the same evidence as grounds for monitoring rather than pre-clearance, and why analysts separate 'decisive' sudden-takeover risk from 'accumulative' societal erosion when deciding what the rules must cover 5.
Coverage across jurisdictions
Historical primacy & cross-jurisdiction tension
First addressed by DoD Responsible AI Strategy and Implementation Pathway on (implicit). Subsequent regimes have either codified, diverged from, or remained silent on this baseline.
- Forum-shoppingExecutive Order 14110 on Safe, Secure, Trustworthy AI↔Executive Order 14179 — Removing Barriers to American Leadership in AI
- Forum-shoppingG7 Hiroshima AI Process Code of Conduct↔Interim Measures for Generative AI Service Management
- Forum-shoppingBletchley Declaration on AI Safety↔OECD AI Principles (Recommendation)
Compare jurisdictions: EU vs US · EU vs UK · EU vs CN
Enforcement & impact
Silent regimes — gap signal
Instruments that do not address Catastrophic & Existential Risk — candidates for future policy work.
- Executive Order 14179 — Removing Barriers to American Leadership in AIUS
- Interim Measures for Generative AI Service ManagementCN
- OECD AI Principles (Recommendation)OECD
- Council of Europe Framework Convention on AIcouncil_of_europe
- India Digital Personal Data Protection Act + AI Advisory (MEITY)IN
- ASEAN Guide on AI Governance and EthicsASEAN
- African Union Continental AI StrategyAfrican_Union
- Singapore Model AI Governance Framework for Generative AISG
- Japan METI AI Guidelines for BusinessJP
- General Data Protection Regulation (GDPR)EU
- OMB Memorandum M-24-10 (Advancing Governance, Innovation, and Risk Management for Agency Use of AI)US
- GSA Generative AI and Specialized Computing Infrastructure Acquisition Resource GuideUS
- FedRAMP AI Cloud Procurement GuidanceUS
- DFARS Subpart 252.204 (Safeguarding Covered Defense Information and Cyber Incident Reporting)US
- California SB 243: Companion ChatbotsUS
- California SB 942: AI Transparency ActUS
- Revised Product Liability Directive (Directive (EU) 2024/2853)EU
- UNESCO Recommendation on the Ethics of Artificial IntelligenceUNESCO
- Directive (EU) 2024/2831 on improving working conditions in platform workEU
- Provisions on the Administration of Deep Synthesis of Internet Information ServicesCN
- TAKE IT DOWN Act (Tools to Address Known Exploitation by Immobilizing Technological Deepfakes on Websites and Networks Act)US
- Italy Law No. 132/2025 on Artificial Intelligence (Legge 23 settembre 2025, n. 132)IT
- Japan AI Promotion Act (Act on the Promotion of Research, Development and Utilization of AI-Related Technologies)JP
See also
Further reading
18 academic & grey-literature sources bearing on this topic — catalogued metadata with a primary link; one-line findings are ✦ AI-generated summaries, labeled as such (charter §7.9). Browse the full literature index.
- Artificial intelligence and synthetic biology: biosecurity risks, dual-use concerns, and governance pathways Peer-reviewed✦ AIReviews biosecurity and dual-use risks at the AI-synthetic-biology interface and maps governance pathways for emerging catastrophic threats.
- Two types of AI existential risk: decisive and accumulative Peer-reviewed✦ AIDistinguishes 'decisive' (sudden takeover) from 'accumulative' AI existential risk, arguing governance must address gradual societal erosion as well as abrupt scenarios.
- Confronting Catastrophic Risk: The International Obligation to Regulate Artificial Intelligence Peer-reviewed✦ AIArgues international law imposes a precautionary-principle obligation on states to regulate AI to mitigate the threat of human extinction.
- Artificial Intelligence and Nuclear Weapons Proliferation: The Technological Arms Race for (In)visibility Peer-reviewed✦ AIAnalyzes how AI-driven detection/concealment in nuclear arsenals reshapes strategic stability and proliferation risk, with governance implications.
- International Agreements on AI Safety: Review and Recommendations for a Conditional AI Safety Treaty Preprint✦ AIProposes a conditional AI safety treaty with a compute threshold triggering mandatory audits by an international network of AI Safety Institutes empowered to halt development if risks are unacceptable.
- Managing extreme AI risks amid rapid progress Peer-reviewed✦ AIWarns "AI safety research is lagging" and present governance initiatives "lack the mechanisms and institutions to prevent misuse and recklessness", urging adaptive governance plus safety R&D.
- Verification methods for international AI agreements Preprint✦ AISurveys '10 verification methods that could detect... unauthorized AI training... and unauthorized data centers', mapping the technical basis for compute-disclosure regimes.
- AI and biosecurity: The need for governance Peer-reviewed✦ AIArgues 'governments should evaluate advanced [biological] models and if needed impose safety measures' to mitigate AI-enabled biosecurity catastrophic risk.
- From Principles to Rules: A Regulatory Approach for Frontier AI Preprint✦ AIRecommends frontier-AI regulation begin with high-level safety principles and migrate to detailed rules (e.g., mandated dangerous-capability evaluations) as regulatory capacity matures.
- Model evaluation for extreme risks Preprint✦ AIProposes "dangerous capability evaluations" and alignment evaluations of frontier models so developers and policymakers can make "responsible decisions about model training, deployment, and security".
- Frontier AI Regulation: Managing Emerging Risks to Public Safety Preprint✦ AIArgues "industry self-regulation is an important first step" but "government intervention will be needed", proposing safety standards, registration and reporting, and compliance mechanisms.
- Can large language models democratize access to dual-use biotechnology? Preprint✦ AIRed-team exercise finding LLM chatbots "may also confer easy access to dual-use technologies capable of inflicting great harm" and could make pandemic-class agents more widely accessible.
- Agentic AI System PreprintYao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K., Cao, Y. (2022), 'ReAct: Synergizing Reasoning and Acting in Language Models.'
- Tool-Use Safety PreprintWallace, E., et al. (2024), 'The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions' (OpenAI) — the canonical industry articulation of instruction-channel hierarchy as a tool-use-safety defence.
- Jailbreak Resistance PreprintZou, A., Wang, Z., Kolter, J. Z., Fredrikson, M. (2023), 'Universal and Transferable Adversarial Attacks on Aligned Language Models' — the canonical demonstration that gradient-based suffix attacks transfer across aligned LLMs.
- Chain-of-Thought Monitoring PreprintKorbak, T., Balesni, M., Barnes, E., Bengio, Y., et al. (2025), 'Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safety.' arXiv:2507.11473.
- Artificial Intelligence Research institute✦ AIUS National Academies' AI consensus-study hub.
- Beyond P(doom) for AI Risk: Quantifying Uncertainty Without Probability Research institute✦ AIArgues AI-risk assessment should characterise structured uncertainty instead of collapsing to a single 'probability of doom' number.
References
Sources cited inline in the analysis (linked from the superscript markers), then the primary instrument sources behind the classifications.
- Shevlane, Farquhar, Garfinkel, et al. (2023) Model evaluation for extreme risks, arXiv. arXiv:2305.15324 — Proposes "dangerous capability evaluations" and alignment evaluations of frontier models so developers and policymakers can make "responsible decisions about model training, deployment, and security". ↩
- Anderljung, Barnhart, Korinek, et al. (2023) Frontier AI Regulation: Managing Emerging Risks to Public Safety, arXiv. arXiv:2307.03718 — Argues "industry self-regulation is an important first step" but "government intervention will be needed", proposing safety standards, registration and reporting, and compliance mechanisms. ↩
- Jonas Schuett, Markus Anderljung, Alexis Carlier, Leonie Koessler, Ben Garfinkel (Centre for the Governance of AI) (2024) From Principles to Rules: A Regulatory Approach for Frontier AI, arXiv (GovAI working paper). arXiv:2407.07300 — Recommends frontier-AI regulation begin with high-level safety principles and migrate to detailed rules (e.g., mandated dangerous-capability evaluations) as regulatory capacity matures. ↩
- Bengio, Hinton, Yao, Song, et al. (2024) Managing extreme AI risks amid rapid progress, Science. 10.1126/science.adn0117 — Warns "AI safety research is lagging" and present governance initiatives "lack the mechanisms and institutions to prevent misuse and recklessness", urging adaptive governance plus safety R&D. ↩
- Atoosa Kasirzadeh (2025) Two types of AI existential risk: decisive and accumulative, Philosophical Studies. 10.1007/s11098-025-02301-3 — Distinguishes 'decisive' (sudden takeover) from 'accumulative' AI existential risk, arguing governance must address gradual societal erosion as well as abrupt scenarios. ↩
- Rebecca Scholefield, Samuel Martin, Otto Barten (2025) International Agreements on AI Safety: Review and Recommendations for a Conditional AI Safety Treaty, arXiv (cs.CY). arXiv:2503.18956 — Proposes a conditional AI safety treaty with a compute threshold triggering mandatory audits by an international network of AI Safety Institutes empowered to halt development if risks are unacceptable. ↩
- arXiv:2210.01790 ↩
- Hubinger, E., et al. (2019), 'Risks from Learned Optimization in Advanced Machine Learning Systems.' Mesa-Optimization. arXiv:1906.01820 — Hubinger, E., et al. (2019), 'Risks from Learned Optimization in Advanced Machine Learning Systems.' ↩
- arXiv:2412.04984 ↩
- arXiv:2412.14093 ↩
- van der Weij, T., Hofstätter, F., Jaffe, O., Brown, S., Ward, F. (2024), 'AI Sandbagging: Language Models can Strategically Underperform on Evaluations.' Sandbagging. arXiv:2406.07358 — van der Weij, T., Hofstätter, F., Jaffe, O., Brown, S., Ward, F. (2024), 'AI Sandbagging: Language Models can Strategically Underperform on Evaluations.' ↩
- Sumeet Ramesh Motwani, Mikhail Baranchuk, Martin Strohmeier, Vijay Bolina, Philip H.S. Torr, Lewis Hammond, Christian Schroeder de Witt (2024) Secret Collusion among AI Agents: Multi-Agent Deception via Steganography, arXiv (NeurIPS 2024). arXiv:2402.07510 — Shows LLM agents can use steganography to communicate covertly, exposing a monitoring/oversight gap for governing multi-agent systems and motivating ongoing mitigation. ↩
- EU-AIA-2024: Art. 51 + Recital 32 — systemic risk overlaps with but does not fully cover catastrophic-risk framing
- US-EO-14110: §4.2(a)(ii) — CBRN + autonomous replication explicitly named
- UK-WHITEPAPER-2023: AISI remit covers frontier-model evaluation; not in white paper text
- G7-HIROSHIMA: Code §1 + §3 — explicit risk-identification including CBRN
- UN-RES-2024: Notes 'shared concerns' but no operative catastrophic-risk text
- NIST-AI-RMF: Map 1.1 risk classification covers catastrophic via 'societal' impact tier; GenAI Profile (2024) adds explicit content
- BLETCHLEY-2023: Declaration §3-5 (substantial risks from frontier AI, including catastrophic harm)
- SEOUL-2024: Frontier AI Safety Commitments §1: identify thresholds for severe risks pre-deployment
- NIST-AI-RMF-GENAI: NIST AI 600-1 §3.1 CBRN Information Uplift; §3.3 Dangerous, Violent, or Hateful Content
- CA-SB-1047: Cal. SB-1047 §22602 — defines 'critical harm' including mass casualties, $500M+ damage
- BR-AIBILL-2024: PL 2338/2023 Art. 14 (excessive-risk AI applications — explicit prohibition + risk-tier framework)
- ANTHROPIC-RSP-2024: RSP v2 §3 — ASL-3 / ASL-4 capability thresholds explicitly target CBRN uplift + autonomous-replication
- OPENAI-PREPAREDNESS-2023: Preparedness Framework risk-tier matrix — Critical tier explicitly targets CBRN, cyber, persuasion, autonomy
- DEEPMIND-FSF-2024: FSF Critical Capability Levels (CCL) — explicit thresholds for autonomy, biosecurity, cyber, persuasion
- META-FRONTIER-2024: Framework critical-risk tier — commit to halt training pre-mitigation if reached
- UK-US-AISI-MOU-2024: Joint evaluation scope encompasses CBRN + autonomy uplift questions; MoU text does not enumerate explicit thresholds
- WH-VOLUNTARY-2023: Commitments §1 references CBRN + bio risks via 'most significant societal risks'; not threshold-explicit
- EU-GPAI-COP-2025: Chapter 3 systemic-risk-tier capability evaluations + serious-incident reporting + model-weight access controls (Art. 55 substrate)
- DOD-RAI-2022: Ethical Principle 'Reliable' + Tenet 4 (Requirements Validation) — JCIDS gating addresses mission-risk; DoDD 3000.09 separately governs autonomy-in-weapons LAWS-specific catastrophic-risk decisions
- CA-SB-53: Bus. & Prof. Code § 22757.11 (definition) operationalized by §§ 22757.12 (framework) + 22757.13 (critical-safety-incident reporting to CalOES)
- NY-RAISE-2025: N.Y. Gen. Bus. Law § 1421(1) requires a large developer to implement and conspicuously publish a written safety and security protocol governing the risk of 'critical harm' from its frontier models, and § 1421(4) requires disclosure of safety incidents within 72 hours; § 1420(7) defines critical harm (100+ deaths/serious injuries or $1B damage via CBRN weapons or autonomous model conduct). NOTE: the floor-text § 1421(2) deployment PROHIBITION was struck by the chapter amendment enacted Mar. 27, 2026 (S8828/A9449), which reoriented the Act to a transparency-and-reporting regime; this cell tracks the RETAINED safety-protocol + incident-reporting duties, not a deployment ban.
- UN-GDC-2024: GDC Objective 5, paras 55(a) and 56(a) (A/RES/79/1, Annex I)
Cite this article 8 formats · BibTeX, RIS, APA, Chicago, … · 1-click copy
Persistent identifier: https://policywindow.org/wiki/catastrophic-risk — committed-stable URL with content-versioning via ?asOf= (rollout pending per methodology §7). DOIs via Zenodo are on the roadmap.
Article tools — track changes, suggest an edit
View history — every captured revision of this article · What links here
22 instruments tracked.
Does governance work? — the social-science evidence
What the peer-reviewed social science shows: whether the harm this topic addresses is empirically real, and whether governance of it works. The badge is the epistemic status of the evidence(not the policy debate) — “thin” or “absent” efficacy evidence is itself a finding (the “second silence”). Each epistemic-status label is Policy Window's editorial assessment of the cited evidence base (a structured classification), not a verdict any single source issues.
The catastrophic-uplift premise is genuinely contested: the empirical uplift studies that exist find current frontier models add little. RAND's red-team study found no statistically significant difference in the viability of bioweapon-attack plans produced with vs. without LLMs (Mouton, Lucas & Guest 2024), and OpenAI's 100-participant trial found GPT-4 gave at most a mild, non-significant accuracy uplift (mean +0.88 out of 10 for PhD experts, +0.25 for students; Patwardhan et al. 2024). Honest caveat: the harm is forward-looking, not yet observed — expert opinion on the catastrophic tail is sharply split (median AI researcher puts ~5% on extremely-bad/extinction outcomes, mean ~9-16% across differently-framed questions, n=2,778; Grace et al. 2024), and forecasters underestimated how fast risk-relevant capabilities (e.g. virology troubleshooting) actually arrived (Forecasting Research Institute 2025), so the relevant capabilities are a moving target rather than a settled magnitude.
Sources: Mouton, Lucas & Guest 2024 (RAND RR-A2977-2, Operational Risks of AI in Large-Scale Biological Attacks: Results of a Red-Team Study); Patwardhan et al. 2024 (OpenAI, Building an Early Warning System for LLM-aided Biological Threat Creation); Grace et al. 2024 (Thousands of AI Authors on the Future of AI, arXiv:2401.02843); Forecasting Research Institute 2025 (Forecasting LLM-enabled Biorisk and the Efficacy of Safeguards)
There is essentially no impact evidence that catastrophic-risk governance reduces catastrophic risk, and structurally there cannot yet be: the harm is a low-probability civilisational tail event, so no controlled trial or before/after evaluation of a realised catastrophe is possible. The dominant instruments are recent, voluntary developer frameworks (Anthropic's Responsible Scaling Policy 2023; OpenAI's Preparedness Framework 2023) built on if-then capability thresholds the developers themselves describe as speculative and qualitative rather than validated risk thresholds. The closest evidence is adjacent and indirect: trained-in deceptive behaviours can persist through standard safety training (Hubinger et al. 2024) — a demonstration that current mitigation may be insufficient, not that any governance regime works — and Anthropic's documented loosening of earlier commitments (RSP 2025 dropped the original pledge to define higher-tier ASL evaluations before developing the corresponding models) illustrates that even the strongest voluntary regimes lack external enforcement or measured efficacy.
Sources: Anthropic 2023 (Responsible Scaling Policy); OpenAI 2023 (Preparedness Framework); Hubinger et al. 2024 (Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training, arXiv:2401.05566); Hendrycks, Mazeika & Woodside 2023 (An Overview of Catastrophic AI Risks, arXiv:2306.12001)