California SB-1047: Safe and Secure Innovation for Frontier AI Models Act
CA-SB-1047 · US
A frontier-model safety-protocol-and-audit bill, not a pre-deployment testing mandate. Passed both chambers (Assembly 28 Aug 2024, Senate concurrence 29 Aug 2024 — the adoptedDate here) and was vetoed by Gov. Newsom on 29 September 2024, so it never became law (status: vetoed; never adopted/enacted). Its core obligation would have required developers of a covered model to adopt a written safety and security protocol and submit a SELF-certified statement of compliance before deployment, taking 'reasonable care' to prevent critical harm; independent THIRD-PARTY audits would have begun only on 1 January 2026 — there was no pre-deployment third-party testing requirement. A 'covered model' was defined conjunctively (>10^26 operations AND >$100M training cost, or fine-tuning >$10M), not a disjunctive trigger. It drew a high-profile coalition — supporters incl. Bengio, Hinton, Musk, Hendrycks and Stuart Russell; opponents incl. Andrew Ng, Fei-Fei Li, Yann LeCun, Pelosi, Lofgren, Khanna, Andreessen Horowitz, Y Combinator and OpenAI. Cited in every 2024-2025 AI governance literature review as the most impactful US state intervention. Currency (2026-06-22): re-introduction did not revive SB 1047; instead author Sen. Wiener's pared-back successor SB 53 (Transparency in Frontier AI Act, tracked here as CA-SB-53) was signed by Gov. Newsom on 2025-09-29 — the first enforceable US state frontier-AI safety law, most provisions effective 2026-01-01.
Vetoed — never took effect
Coverage cells below reflect this instrument's operative content as it would have applied had it been enacted (it was vetoed and never took effect). Time-sensitive policy briefs should also cite the source document directly and check for amendments. PW does not track legislative-progress updates within a single catalog snapshot.
Background & scope
California SB-1047: Safe and Secure Innovation for Frontier AI Models Act addresses 4 contested AI-governance topics explicitly, 2 via general principles.
Provisions & coverage
- governsFoundation Models / GPAICal. SB-1047 §22602 — 'covered model' = trained with >10^26 operations AND >$100M cost (or fine-tuning >$10M); vetoed 29 Sep 2024[12]
- governsCompute-Threshold ReportingCal. SB-1047 §22603(b) — annual reporting of training compute + safety determination[12]
- implicitTransparency ObligationsRequired safety determinations are public; full safety case is to regulator only[12]
- implicitIndividual RedressWhistleblower protections (§22607) + AG enforcement (§22608); no individual redress[12]
- governsCatastrophic & Existential RiskCal. SB-1047 §22602 — defines 'critical harm' including mass casualties, $500M+ damage[12]
- governsOpen-Weight Frontier ReleaseVetoed bill — would have required covered models (incl. open-weight releases) to adopt a safety & security protocol + self-certified compliance, with independent third-party audits from 2026 (Anthropic + Meta objected on different grounds)[12]
Operative Mechanics: A Compute-and-Cost Threshold Never Brought Into Force
SB-1047 passed both chambers of the California legislature in late August 2024 (Assembly 28 August, Senate concurrence 29 August) but was vetoed by Governor Newsom on 29 September 2024, so its mechanics describe a regime that never took effect. The bill targeted a 'covered model' defined in § 22602 conjunctively — trained using more than 10^26 integer or floating-point operations AND costing more than $100M (or fine-tuned at a cost above $10M) — not a disjunctive compute-or-cost trigger. Its central obligation was not pre-deployment third-party testing: a developer had to adopt a written safety and security protocol before training and submit an annual SELF-certified statement of compliance, exercising 'reasonable care' to prevent 'critical harm'; independent third-party audits were to begin only on 1 January 2026 (§ 22603). 'Critical harm' (§ 22602(g)) was defined across distinct categories whose thresholds differ: creation or use of a chemical, biological, radiological or nuclear weapon causing mass casualties (no dollar floor), and cyberattacks on critical infrastructure or autonomous criminal acts causing either mass casualties OR at least $500M in damage — so the $500M figure attaches specifically to the cyber and autonomous-crime categories, not to the CBRN category. Enforcement ran through the Attorney General (§ 22606) plus whistleblower protections (§ 22607), with no private right of action. Compute as the regulatory hook reflects the argument that it is uniquely 'detectable, excludable, and quantifiable' 1.
Cross-Jurisdiction Position: A US State Mirror of EU Systemic-Risk Logic
As the first US state-level frontier-model safety bill, SB-1047 paralleled the EU AI Act's general-purpose/systemic-risk tier, which itself sits atop unstable terminology — Fernández-Llorca et al. (2025) trace the drift across 'AI system, general purpose AI system, foundation model, and generative AI' 2, and Hulok (2025) notes such models challenge 'authorship, accountability, and control' 3. Where Brussels relied on a fixed 10^25 FLOP presumption, California paired a 10^26 threshold (§ 22603) with a $100M cost limb, hedging against compute alone. Lacking the EU's GDPR substrate — relevant given foundation models that 'memorize and leak' training data 4 — the bill leaned wholly on catastrophic-harm framing rather than data-protection grounding.
Key Fault Lines: Threshold Evasion, Redress Gaps, and the Veto Rationale
Critiques clustered on three axes. First, compute thresholds are gameable: Pistillo and Villalobos (2025) document 'enhancement techniques... capable of decreasing training compute usage while preserving... model capabilities' 5, so a $100M/10^26 trigger (§ 22603) risks under-capturing capable models. Second, redress was thin — § 22607–22608 offered whistleblower and AG enforcement but no individual remedy, against literature finding decision subjects need genuinely 'meaningful' contestation 67. Third, Newsom's veto argued the threshold was a poor proxy that could miss smaller-but-dangerous systems while burdening frontier labs — a critique sharpened by accounts of biosecurity dual-use risk 8 that compute gates address only obliquely. These fault lines played out as an unusually public coalition fight: prominent researchers backed the bill (Yoshua Bengio, Geoffrey Hinton, Stuart Russell, Dan Hendrycks) alongside Elon Musk, while a comparably prominent bloc opposed it — Andrew Ng, Fei-Fei Li and Yann LeCun on the technical side, members of California's congressional delegation (Pelosi, Lofgren, Khanna), and industry actors including Andreessen Horowitz, Y Combinator and OpenAI — a split that made the veto as much a political-economy outcome as a technical judgement.
Implementation Trajectory: From Veto to the Enforceable SB-53 Successor
Although vetoed, SB-1047 is cited in nearly every 2024–2025 AI governance review as the most consequential US state intervention, shaping the agenda even in defeat. Re-introduction did not revive the original; instead author Sen. Wiener's pared-back successor, SB-53 (Transparency in Frontier AI Act), was signed by Newsom on 29 September 2025 — the first enforceable US state frontier-AI safety law, most provisions effective 1 January 2026 (Office of Governor Newsom 2025). The trajectory tracks the catastrophic-risk scholarship that animated the bill: warnings that governance 'lacks the mechanisms and institutions to prevent misuse and recklessness' 9, the decisive-versus-accumulative risk split 10, and proposed compute-triggered audit treaties 11.
Enforcement & impact
Cross-jurisdiction comparison
How peer instruments treat the topics California SB-1047: Safe and Secure Innovation for Frontier AI Models Act governs.
| Topic | EU-AIA-2024 | US-EO-14110 | US-EO-14179 | UK-WHITEPAPER-2023 | CN-GENAI-2023 | G7-HIROSHIMA | OECD-AI-PRIN | COE-AI-CONV | UN-RES-2024 | NIST-AI-RMF | BLETCHLEY-2023 | SEOUL-2024 | NIST-AI-RMF-GENAI | IN-DPDP-2023 | BR-AIBILL-2024 | ASEAN-AI-GUIDE-2024 | AU-AI-STRATEGY-2024 | ANTHROPIC-RSP-2024° | OPENAI-PREPAREDNESS-2023° | DEEPMIND-FSF-2024° | META-FRONTIER-2024° | UK-US-AISI-MOU-2024 | WH-VOLUNTARY-2023 | SG-MODEL-AI-2024 | JP-METI-AI-2024 | EU-GDPR-2016 | EU-GPAI-COP-2025 | OMB-M-24-10 | GSA-AI-GUIDE-2024 | DOD-RAI-2022 | FEDRAMP-AI-2024 | DFARS-252-204 | CA-SB-53 | CA-SB-243 | CA-SB-942 | EU-PLD-2024 | UNESCO-AI-ETHICS-2021 | EU-PWD-2024 | CN-DEEPSYN-2022 | NY-RAISE-2025 | US-TAKEITDOWN-2025 | IT-AILAW-2025 | JP-AIPROMO-2025 | UN-GDC-2024 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Foundation Models / GPAI | governs | governs | silent | implicit | governs | governs | implicit | implicit | silent | governs | governs | governs | governs | implicit | governs | implicit | silent | governs | governs | governs | governs | governs | governs | governs | governs | silent | governs | implicit | governs | implicit | implicit | implicit | governs | silent | implicit | silent | silent | silent | silent | governs | silent | silent | implicit | implicit |
| Compute-Threshold Reporting | governs | governs | silent | silent | silent | silent | silent | silent | silent | silent | implicit | implicit | silent | silent | silent | silent | silent | implicit | implicit | silent | silent | silent | implicit | silent | silent | silent | silent | governs | governs | implicit | implicit | implicit | implicit | silent | silent | silent | silent | silent | silent | implicit | silent | silent | implicit | silent |
| Catastrophic & Existential Risk | implicit | governs | silent | implicit | silent | governs | silent | silent | implicit | implicit | governs | governs | governs | silent | governs | silent | silent | governs | governs | governs | governs | implicit | implicit | silent | silent | silent | governs | silent | silent | implicit | silent | silent | governs | silent | silent | silent | silent | silent | silent | governs | silent | silent | silent | implicit |
| Open-Weight Frontier Release | governs | implicit | silent | silent | implicit | silent | silent | silent | silent | silent | silent | implicit | silent | silent | silent | silent | implicit | implicit | implicit | implicit | governs | silent | silent | silent | silent | silent | silent | silent | silent | silent | silent | silent | silent | silent | governs | silent | silent | silent | silent | silent | silent | silent | silent | implicit |
°= industry self-imposed voluntary framework. Comparing a voluntary code's "governs" tint with a binding regulation's "governs" tint flattens the legal-force distinction; use the instrument-page banner for the operative status of each.
See also
Per-audience views
- Provisions →Article-by-article obligation breakdown for procurement + RFP authors.
- Disclosure form →Vendor-disclosure questionnaire derived from this instrument's operative obligations.
- Harm narratives →Documented harms relevant to this instrument's topics, for civil-society advocacy.
- Briefing pack →Journalist-ready summary with quotes + dates + primary-source links.
Article tools — track changes, suggest an edit
View history — every captured revision of this article · What links here
Further reading
102 academic & grey-literature sources on the topics this instrument addresses (not commentary on the instrument itself) — catalogued metadata with a primary link; one-line findings are ✦ AI-generated summaries, labeled as such (charter §7.9). Browse the full literature index.
- Artificial intelligence and synthetic biology: biosecurity risks, dual-use concerns, and governance pathways Peer-reviewed✦ AIReviews biosecurity and dual-use risks at the AI-synthetic-biology interface and maps governance pathways for emerging catastrophic threats.
- An interdisciplinary account of the terminological choices by EU policymakers ahead of the final agreement on the AI Act: AI system, general purpose AI system, foundation model, and generative AI Peer-reviewed✦ AITraces how the AI Act's legal text shifted across versions among the terms 'AI system, general purpose AI system, foundation model, and generative AI', exposing definitional instability in the regime.
- The EU model of AI governance: regulating artificial intelligence through law and policy Peer-reviewed✦ AIAnalyses how the AI Act's risk-based model handles general-purpose and foundation models whose 'autonomous content generation challenges legal categories of authorship, accountability, and control'.
- Generative AI and data protection Peer-reviewed✦ AIExamines friction between foundation-model training and the GDPR, noting models that 'memorize and leak pieces of training data' cannot be treated as anonymous.
- Defending Compute Thresholds Against Legal Loopholes Preprint✦ AIIdentifies 'enhancement techniques that are capable of decreasing training compute usage while preserving... model capabilities', exposing loopholes in compute-reporting thresholds.
- Identifying Algorithmic Decision Subjects' Needs for Meaningful Contestability Peer-reviewed✦ AIEmpirically elicits what decision subjects need for contestation to be 'meaningful', informing the design of effective remedies and appeal mechanisms for ADM.
- Two Means to an End Goal: Connecting Explainability and Contestability in the Regulation of Public Sector AI Preprint✦ AIInterview study with 14 regulation experts distinguishes judicial vs non-judicial and individual vs collective contestation channels for public-sector AI remedies.
- Two types of AI existential risk: decisive and accumulative Peer-reviewed✦ AIDistinguishes 'decisive' (sudden takeover) from 'accumulative' AI existential risk, arguing governance must address gradual societal erosion as well as abrupt scenarios.
- Confronting Catastrophic Risk: The International Obligation to Regulate Artificial Intelligence Peer-reviewed✦ AIArgues international law imposes a precautionary-principle obligation on states to regulate AI to mitigate the threat of human extinction.
- Artificial Intelligence and Nuclear Weapons Proliferation: The Technological Arms Race for (In)visibility Peer-reviewed✦ AIAnalyzes how AI-driven detection/concealment in nuclear arsenals reshapes strategic stability and proliferation risk, with governance implications.
- International Agreements on AI Safety: Review and Recommendations for a Conditional AI Safety Treaty Preprint✦ AIProposes a conditional AI safety treaty with a compute threshold triggering mandatory audits by an international network of AI Safety Institutes empowered to halt development if risks are unacceptable.
- Managing extreme AI risks amid rapid progress Peer-reviewed✦ AIWarns "AI safety research is lagging" and present governance initiatives "lack the mechanisms and institutions to prevent misuse and recklessness", urging adaptive governance plus safety R&D.
+ 90 more across this instrument's topics — see the literature index.
References
Sources cited inline in the analysis (linked from the superscript markers), then the primary instrument sources behind the classifications.
- Sastry, Heim, Belfield, Anderljung, Brundage, et al. (2024) Computing Power and the Governance of Artificial Intelligence, arXiv. arXiv:2402.08797 — Argues compute is a uniquely governable lever because it is "detectable, excludable, and quantifiable, and is produced via an extremely concentrated supply chain". ↩
- David Fernández-Llorca, Emilia Gómez, Ignacio Sánchez, Gabriele Mazzini (2025) An interdisciplinary account of the terminological choices by EU policymakers ahead of the final agreement on the AI Act: AI system, general purpose AI system, foundation model, and generative AI, Artificial Intelligence and Law. 10.1007/s10506-024-09412-y — Traces how the AI Act's legal text shifted across versions among the terms 'AI system, general purpose AI system, foundation model, and generative AI', exposing definitional instability in the regime. ↩
- Martina Hulok (2025) The EU model of AI governance: regulating artificial intelligence through law and policy, ERA Forum. 10.1007/s12027-025-00869-1 — Analyses how the AI Act's risk-based model handles general-purpose and foundation models whose 'autonomous content generation challenges legal categories of authorship, accountability, and control'. ↩
- Hannah Ruschemeier (2025) Generative AI and data protection, Cambridge Forum on AI: Law and Governance. 10.1017/cfl.2024.2 — Examines friction between foundation-model training and the GDPR, noting models that 'memorize and leak pieces of training data' cannot be treated as anonymous. ↩
- Matteo Pistillo, Pablo Villalobos (2025) Defending Compute Thresholds Against Legal Loopholes, arXiv (cs.CY). arXiv:2502.00003 — Identifies 'enhancement techniques that are capable of decreasing training compute usage while preserving... model capabilities', exposing loopholes in compute-reporting thresholds. ↩
- Mireia Yurrita, Himanshu Verma, Agathe Balayn, Kars Alfrink, Ujwal Gadiraju, and Alessandro Bozzon (2025) Identifying Algorithmic Decision Subjects' Needs for Meaningful Contestability, Proceedings of the ACM on Human-Computer Interaction (CSCW). 10.1145/3757415 — Empirically elicits what decision subjects need for contestation to be 'meaningful', informing the design of effective remedies and appeal mechanisms for ADM. ↩
- arXiv:2504.18236 ↩
- Kirolos Eskandar (2026) Artificial intelligence and synthetic biology: biosecurity risks, dual-use concerns, and governance pathways, AI and Ethics (Springer). 10.1007/s43681-025-00872-9 — Reviews biosecurity and dual-use risks at the AI-synthetic-biology interface and maps governance pathways for emerging catastrophic threats. ↩
- Bengio, Hinton, Yao, Song, et al. (2024) Managing extreme AI risks amid rapid progress, Science. 10.1126/science.adn0117 — Warns "AI safety research is lagging" and present governance initiatives "lack the mechanisms and institutions to prevent misuse and recklessness", urging adaptive governance plus safety R&D. ↩
- Atoosa Kasirzadeh (2025) Two types of AI existential risk: decisive and accumulative, Philosophical Studies. 10.1007/s11098-025-02301-3 — Distinguishes 'decisive' (sudden takeover) from 'accumulative' AI existential risk, arguing governance must address gradual societal erosion as well as abrupt scenarios. ↩
- Rebecca Scholefield, Samuel Martin, Otto Barten (2025) International Agreements on AI Safety: Review and Recommendations for a Conditional AI Safety Treaty, arXiv (cs.CY). arXiv:2503.18956 — Proposes a conditional AI safety treaty with a compute threshold triggering mandatory audits by an international network of AI Safety Institutes empowered to halt development if risks are unacceptable. ↩
- Cal. SB-1047 (Wiener, 2024)
- Cal. SB-1047 §22602 — 'covered model' = trained with >10^26 operations AND >$100M cost (or fine-tuning >$10M); vetoed 29 Sep 2024
- Cal. SB-1047 §22603(b) — annual reporting of training compute + safety determination
- Required safety determinations are public; full safety case is to regulator only
- Whistleblower protections (§22607) + AG enforcement (§22608); no individual redress
- Cal. SB-1047 §22602 — defines 'critical harm' including mass casualties, $500M+ damage
- Vetoed bill — would have required covered models (incl. open-weight releases) to adopt a safety & security protocol + self-certified compliance, with independent third-party audits from 2026 (Anthropic + Meta objected on different grounds)
How to cite this article
Cite this article 8 formats · BibTeX, RIS, APA, Chicago, … · 1-click copy
Persistent identifier: https://policywindow.org/wiki/ca-sb-1047 — committed-stable URL with content-versioning via ?asOf= (rollout pending per methodology §7). DOIs via Zenodo are on the roadmap.
Does this instrument’s approach work? — the social-science evidence
Aggregated over the 6 topics this instrument governs: whether each harm is empirically real, and whether the peer-reviewed evidence shows governance reduces it. The badge is the epistemic status of the evidence— “thin”/“absent” efficacy evidence is itself a finding (the “second silence”). Each epistemic-status label is Policy Window's editorial assessment of the cited evidence base (a structured classification), not a verdict any single source issues.
Of the 6 governed topics with a social-science evidence review, evidence that governance reduces the harm is established for 0, contested for 0, thin for 0, and absent for 6 — for most, no replicated study yet shows this instrument's approach works (the "second silence").
Catastrophic & Existential Risk
The catastrophic-uplift premise is genuinely contested: the empirical uplift studies that exist find current frontier models add little. RAND's red-team study found no statistically significant difference in the viability of bioweapon-attack plans produced with vs. without LLMs (Mouton, Lucas & Guest 2024), and OpenAI's 100-participant trial found GPT-4 gave at most a mild, non-significant accuracy uplift (mean +0.88 out of 10 for PhD experts, +0.25 for students; Patwardhan et al. 2024). Honest caveat: the harm is forward-looking, not yet observed — expert opinion on the catastrophic tail is sharply split (median AI researcher puts ~5% on extremely-bad/extinction outcomes, mean ~9-16% across differently-framed questions, n=2,778; Grace et al. 2024), and forecasters underestimated how fast risk-relevant capabilities (e.g. virology troubleshooting) actually arrived (Forecasting Research Institute 2025), so the relevant capabilities are a moving target rather than a settled magnitude.
Sources: Mouton, Lucas & Guest 2024 (RAND RR-A2977-2, Operational Risks of AI in Large-Scale Biological Attacks: Results of a Red-Team Study); Patwardhan et al. 2024 (OpenAI, Building an Early Warning System for LLM-aided Biological Threat Creation); Grace et al. 2024 (Thousands of AI Authors on the Future of AI, arXiv:2401.02843); Forecasting Research Institute 2025 (Forecasting LLM-enabled Biorisk and the Efficacy of Safeguards)
There is essentially no impact evidence that catastrophic-risk governance reduces catastrophic risk, and structurally there cannot yet be: the harm is a low-probability civilisational tail event, so no controlled trial or before/after evaluation of a realised catastrophe is possible. The dominant instruments are recent, voluntary developer frameworks (Anthropic's Responsible Scaling Policy 2023; OpenAI's Preparedness Framework 2023) built on if-then capability thresholds the developers themselves describe as speculative and qualitative rather than validated risk thresholds. The closest evidence is adjacent and indirect: trained-in deceptive behaviours can persist through standard safety training (Hubinger et al. 2024) — a demonstration that current mitigation may be insufficient, not that any governance regime works — and Anthropic's documented loosening of earlier commitments (RSP 2025 dropped the original pledge to define higher-tier ASL evaluations before developing the corresponding models) illustrates that even the strongest voluntary regimes lack external enforcement or measured efficacy.
Sources: Anthropic 2023 (Responsible Scaling Policy); OpenAI 2023 (Preparedness Framework); Hubinger et al. 2024 (Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training, arXiv:2401.05566); Hendrycks, Mazeika & Woodside 2023 (An Overview of Catastrophic AI Risks, arXiv:2306.12001)
Compute-Threshold Reporting
Whether training-compute (FLOP) is a defensible proxy for governance-relevant capability is genuinely contested in the literature. The strongest empirical pressure against it is algorithmic efficiency: Ho, Besiroglu, Erdil et al. (2024) estimate the compute needed to reach a fixed language-model performance level has halved roughly every eight months (95% CI ~5-14 months, i.e. ~3x/year), so any static FLOP-to-capability mapping decays quickly; Hooker (2024) argues FLOP measures operations rather than end-performance, since techniques such as fine-tuning, retrieval, chain-of-thought and tool use can add large capability gains without proportional training compute, and Ord (2025) shows inference-time scaling further decouples deployed capability from training compute. Honest caveat: defenders (Heim & Koessler 2024; Pilz, Heim & Brown 2025) note compute remains the most quantifiable, externally verifiable, and ex-ante measurable correlate of frontier capability currently available, while themselves conceding it is an imperfect proxy that should not be used in isolation — the disagreement is about durability and precision, not whether any correlation exists.
Sources: Ho, Besiroglu, Erdil, Owen, Rahman, Guo, Atkinson, Thompson & Sevilla 2024, Algorithmic progress in language models, NeurIPS 2024 (arXiv:2403.05812; Epoch AI); Hooker 2024, On the Limitations of Compute Thresholds as a Governance Strategy (arXiv:2407.05694); Ord 2025, Inference Scaling Reshapes AI Governance (arXiv:2503.05705); Heim & Koessler 2024, Training Compute Thresholds: Features and Functions in AI Regulation (arXiv:2405.10799); Pilz, Heim & Brown 2025, Increased Compute Efficiency and the Diffusion of AI Capabilities (AAAI 2025; arXiv:2311.15377)
There is no rigorous evidence that compute-threshold reporting reduces harm or achieves its stated aim, because the regimes have not produced an evaluable record. The US 10^26-FLOP reporting obligation (Executive Order 14110, invoking the Defense Production Act) was revoked on 20 January 2025 (by EO 14148) before its recurring binding reporting rule was finalized — the implementing BIS notice of proposed rulemaking (Sept 2024) never took effect, so no durable reporting record materialized; and the EU AI Act's 10^25-FLOP systemic-risk obligations for general-purpose models only became applicable on 2 August 2025 (with transitional periods into 2027), so no outcome evaluation yet exists. Moreover the 10^25 figure is a rebuttable presumption sitting alongside qualitative high-impact criteria (Art. 51(1)(a) and (2), rebuttable under Art. 52(2)), not a validated risk cutoff. The closest analogue is the broader regulatory-disclosure-mandate literature (Fung, Graham & Weil 2007), which documents that transparency policies' effects on outcomes are highly heterogeneous and frequently ineffective or counterproductive absent enforcement and downstream use — implying that the reporting trigger working as intended is an open empirical question, not a documented result.
Sources: U.S. Executive Order 14110 (2023), Sec. 4.2 (10^26 FLOP, Defense Production Act); revoked by Executive Order 14148 (Jan 20, 2025); EU AI Act, Reg. (EU) 2024/1689, Art. 51 (10^25 FLOP systemic-risk rebuttable presumption; applicable Aug 2, 2025); Fung, Graham & Weil 2007, Full Disclosure: The Perils and Promise of Transparency (Cambridge University Press)
Foundation Models / GPAI
Whether the foundation-model category maps to a coherent capability/risk tier is genuinely contested. The original case rests on scale-driven 'emergent abilities' that appear unpredictably above a size threshold (Wei et al. 2022; Ganguli et al. 2022 documented capabilities that are smoothly predictable in aggregate loss yet locally surprising), but Schaeffer, Miranda & Koyejo (2023, a NeurIPS Outstanding Paper) showed many 'emergent' jumps are artefacts of discontinuous metrics and dissolve under linear/continuous scoring — implying capability scales more smoothly than a sharp tier would suggest. Honest caveat: this is a live empirical disagreement about measurement, not a settled finding either way, and compute (the regulatory proxy) is an imperfect stand-in for capability or risk regardless of which side is right.
Sources: Wei et al. 2022 (Emergent Abilities of Large Language Models, TMLR; arXiv:2206.07682); Schaeffer, Miranda & Koyejo 2023 (Are Emergent Abilities of Large Language Models a Mirage?, NeurIPS 2023, Outstanding Paper; arXiv:2304.15004); Ganguli et al. 2022 (Predictability and Surprise in Large Generative Models, ACM FAccT; DOI 10.1145/3531146.3533229)
There is no impact evaluation showing that GPAI/foundation-model governance reduces harm — the rules are too new (EU AI Act GPAI obligations and the 10^25-FLOP systemic-risk presumption only began binding on 2 August 2025) and the central regulatory lever is itself contested: Hooker (2024) argues compute thresholds are a shortsighted proxy because compute does not reliably track capability or risk, and the thresholds already diverge across jurisdictions (EU 10^25 vs. the now-rescinded US EO 14110's 10^26 operations, rescinded 20 January 2025). The mandated mitigation methods also lack validated efficacy: model evaluation and red-teaming face well-documented coverage limits and an 'audit gap' in the survey/position literature (behavioural testing cannot establish the absence of untested failure modes), and adversarial red-teaming repeatedly defeats deployed safeguards — the UK AI Safety Institute reports finding universal jailbreaks for every frontier system it has tested, and a large public agent-injection competition elicited policy violations across all 22 frontier models tested from ~1.8M attacks (Zou et al. 2025). Even compliant evaluation therefore cannot yet certify the safety the rules demand. (Caveat: this is an absence-of-evidence claim — no efficacy study has been done — not evidence the rules are ineffective.)
Sources: Hooker 2024 (On the Limitations of Compute Thresholds as a Governance Strategy, arXiv:2407.05694); EU AI Act Arts. 51 & 55 (GPAI systemic-risk presumption, 10^25 FLOP; binding 2 Aug 2025); US EO 14110 (10^26-operation reporting threshold, rescinded 20 Jan 2025 by EO 14148); Zou et al. 2025 (Security Challenges in AI Agent Deployment: Insights from a Large Scale Public Competition / Gray Swan Arena, arXiv:2507.20526 — 22 frontier agents, ~1.8M attacks); UK AI Safety/Security Institute, Frontier AI Trends Report (universal jailbreaks for every system tested); METR, Common Elements of Frontier AI Safety Policies (2024)
Open-Weight Frontier Release
The empirical picture splits into two well-separated questions. (1) The MECHANISM that distinguishes open-weight release — that safety guardrails can be cheaply and irreversibly stripped once weights are public — is established: Qi et al. (2024) removed GPT-3.5 Turbo safety alignment by fine-tuning on only ~10 adversarially designed examples for under $0.20 (and the attack generalizes to Llama-2), and even purpose-built tamper-resistant safeguards (Tamirisa et al. 2025, TAR) were subsequently shown to be defeatable by adaptive fine-tuning (Qi et al. 2024, durability critique). (2) Whether this mechanism produces real-world CATASTROPHIC uplift is genuinely contested and, for the headline biosecurity case, currently unsupported: RAND's red-team study found no statistically significant difference in the viability of bioweapon attack plans produced with versus without LLM assistance (Mouton, Lucas & Guest 2024), and OpenAI's 100-participant trial found at most mild uplift over an internet baseline (Patwardhan et al. 2024). Honest caveat: these null/mild results are time-stamped to 2023-2024 frontier capability and to biothreats specifically; the marginal-risk framework (Kapoor, Bommasani et al. 2024) concludes the evidence base is too thin to characterize marginal risk across most misuse vectors, so 'no measured harm yet' is not 'no harm.'
Sources: Kapoor, Bommasani, Klyman, Longpre et al. 2024, 'Position: On the Societal Impact of Open Foundation Models', PMLR 235 / ICML 2024 (arXiv 2403.07918); Mouton, Lucas & Guest 2024, RAND RR-A2977-2, 'The Operational Risks of AI in Large-Scale Biological Attacks: Results of a Red-Team Study'; Qi, Zeng, Xie, Chen, Jia, Mittal & Henderson 2024, 'Fine-tuning Aligned Language Models Compromises Safety', ICLR 2024 (arXiv 2310.03693); Tamirisa et al. 2025, 'Tamper-Resistant Safeguards for Open-Weight LLMs', ICLR 2025 (arXiv 2408.00761); Qi, Wei, Carlini, Huang, Xie, He, Jagielski, Nasr, Mittal & Henderson 2024, 'On Evaluating the Durability of Safeguards for Open-Weight LLMs' (arXiv 2412.07097); Patwardhan et al. 2024, 'Building an early warning system for LLM-aided biological threat creation', OpenAI
There is no impact evaluation showing that any specific weight-release governance regime reduces downstream harm, because no binding regime has been implemented and measured: California SB-1047's release-conditioning framework was vetoed in September 2024, and the EU AI Act's open-source carve-outs (Recital 102, Art. 53(2)) exempt most open-weight models (those below the systemic-risk compute threshold) from the documentation obligations that would generate evaluable conduct. The structural obstacle is also documented: Kapoor, Bommasani et al. (2024) characterize open-weight release as effectively irreversible and poorly monitorable once weights are public, so post-release governance has little to act on. The closest analogue evidence — technology export controls — is mixed and points to circumvention: commentators argue blanket export controls on freely copyable open-source models cannot work (Just Security 2024), and independent analyses of the post-2022 semiconductor controls document displacement to less-regulated channels (smuggling, threshold-tuned chip variants, cloud access) rather than disappearance of activity (e.g., CSIS, FPRI 2024), suggesting recipient-restriction regimes face the same leakage problem for weights. (Caveat: this is analogical, not direct evidence about weight-release governance, which remains unmeasured.)
Sources: Kapoor, Bommasani, Klyman, Longpre et al. 2024, 'Position: On the Societal Impact of Open Foundation Models', PMLR 235 (arXiv 2403.07918); California SB-1047 (2024, vetoed by Gov. Newsom 29 Sep 2024); EU AI Act Regulation (EU) 2024/1689, Recital 102 & Art. 53(2) open-source exemptions; Just Security 2024, 'Export Controls on Open-Source Models Will Not Win the AI Race'; CSIS, 'The Limits of Chip Export Controls in Meeting the China Challenge' and FPRI 2024, 'Breaking the Circuit: US-China Semiconductor Controls' (export-control circumvention analogue)
Individual Redress
The premise behind redress — that affected people lack meaningful recourse against automated decisions — is real, but the flagship instrument is weaker than commonly assumed. Wachter, Mittelstadt & Floridi (2017) show GDPR creates only a limited 'right to be informed,' not a binding 'right to explanation' of specific decisions; and controlled work finds the explanations actually delivered do not measurably improve lay decision accuracy over showing the bare AI prediction (Alufaisan et al. 2021; and a 2022 meta-analysis by Schemmer et al. — screening 393 articles down to 9 in the final analysis — reports 'no effect of explanations on users' performance compared to sole AI predictions,' even though XAI overall had a positive effect). Honest caveat: the legitimacy/dignity value of being heard is empirically well established in the procedural-justice tradition even where outcome accuracy is unchanged, so 'redress fails' depends on which aim is measured.
Sources: Wachter, Mittelstadt & Floridi 2017 (International Data Privacy Law 7(2):76); Alufaisan, Marusich, Bakdash, Zhou & Kantarcioglu 2021 (Proceedings of the AAAI Conference on AI 35(8):6618); Schemmer, Hemmer, Nitsche, Kühl & Vössing 2022 (AAAI/ACM AIES '22, meta-analysis)
There is no rigorous impact evaluation showing that mandated redress mechanisms (right-to-explanation, appeal, human-in-the-loop review) actually reduce erroneous or unfair automated decisions — the evidence that the rule works is itself missing. The closest experimental analogues are discouraging: explanations increase humans' acceptance of AI recommendations regardless of correctness (Bansal et al. 2021), and algorithm-in-the-loop oversight can introduce racial disparities and exhibit automation bias rather than reliably catching model errors (Green & Chen 2019). The procedural-justice literature (Tyler 1990; Lind & Tyler 1988) robustly supports a legitimacy and compliance benefit of fair process, but it measures perceived fairness, not reduction of the substantive decision harm redress is meant to cure.
Sources: Bansal, Wu, Zhou, Fok, Nushi, Kamar, Ribeiro & Weld 2021 (CHI '21); Green & Chen 2019 (Disparate Interactions, ACM FAT* '19); Tyler 1990 (Why People Obey the Law, Yale Univ. Press); Lind & Tyler 1988 (The Social Psychology of Procedural Justice, Plenum Press)
Transparency Obligations
Documentation artifacts (model cards, datasheets) are well-specified as proposals and are genuinely adopted, but the empirical premise that mandated disclosure produces meaningful transparency is contested. Selbst & Barocas (2018) argue inscrutability and non-intuitiveness are distinct problems and that disclosing rules does not resolve the latter, and large-scale audits find documentation is sparsely and unevenly completed: a systematic analysis of 32,111 Hugging Face model cards (Liang et al. 2024) found environmental-impact, limitations and evaluation sections least often filled, and Bhat et al. (2023, 45 practitioners) found a substantial gap between the documentation proposal and actual practice. Honest caveat: the documentation frameworks themselves are real and adopted, so the dispute is about whether disclosure conveys decision-relevant information, not whether the artifacts exist.
Sources: Selbst & Barocas 2018 (Fordham Law Review 87:1085-1139); Liang et al. 2024 (Nature Machine Intelligence, s42256-024-00857-z, 'Systematic analysis of 32,111 AI model cards'); Bhat et al. 2023 (CHI '23, 'Aspirations and Practice of ML Model Documentation', DOI 10.1145/3544548.3581518); Mitchell et al. 2019 (FAccT, Model Cards for Model Reporting); Gebru et al. 2021 (CACM 64(12):86-92, Datasheets for Datasets)
There is no rigorous impact evaluation showing that AI transparency mandates (model cards, training-data summaries) measurably reduce bias, misuse or accidents — the central regulatory assumption is empirically untested, partly because flagship mandates like EU AI Act Art. 53(1)(d) GPAI training-data summaries are only subject to AI Office enforcement/verification from 2 August 2026 (the obligation itself began 2 August 2025 for new models). The closest analogue, mandated consumer disclosure, shows small and context-dependent effects: Bollinger, Leslie & Sorensen (2011) found mandatory calorie posting cut average calories per transaction by about 6%, while Loewenstein, Sunstein & Golman (2014) review evidence that disclosure effects are frequently diminished or even reversed by limited attention and often change provider rather than recipient behavior. These are analogues, not AI studies; no study demonstrates that AI transparency disclosure achieves its stated downstream safety aims.
Sources: Bollinger, Leslie & Sorensen 2011 (AEJ: Economic Policy 3(1):91-128); Loewenstein, Sunstein & Golman 2014 (Annual Review of Economics 6:391-419, 'Disclosure: Psychology Changes Everything'); EU AI Act Art. 53(1)(d) GPAI training-data summary (obligation from 2 Aug 2025; AI Office enforcement from 2 Aug 2026)