Open Problems in AI Governance

Ranked agenda

Scored on importance, neglectedness, difficulty, action-relevance, robustness across timelines, and national+international relevance. The top three are preconditions for the rest: define the object, define legitimate authority, define evidentiary standards.

1. 1Epistemic burden of proofUnder deep uncertainty, who bears the burden of proof: developers to show sufficient safety, or regulators to show sufficient danger?
2. 2Governable objectShould governance target models, systems, deployments, compute, data, companies, capabilities, behaviours, supply chains, or socio-technical arrangements?
3. 3LegitimacyWho should decide whether powerful AI systems may be trained, deployed, paused, nationalised, restricted, open-sourced, or integrated into public infrastructure?
4. 4Verification without LeviathanHow can governments or international institutions verify claims about training runs, compute use, model weights, security, evaluations, and deployment restrictions without excessive surveillance, centralisation, or abuse?
5. 5Evaluation and assuranceAre model evaluations, red-teaming, benchmarks, audits, and safety cases reliable enough to determine whether a system should be trained, deployed, restricted, or recalled?
6. 6Thresholds and tripwiresWhich thresholds should trigger reporting, licensing, external evaluation, compute controls, restricted deployment, model-weight security requirements, or a pause?
7. 7Race stabilityUnder what conditions can firms and states cooperate on AI safety rather than race toward deployment?
8. 8AGI transition and post-AGI constitutionalismWhat institutions, rules, and technical controls are needed if AI systems begin to exceed human experts across most cognitive tasks, accelerate AI R&D, autonomously act in the world, or become difficult to control?
9. 9Corporate governanceCan corporate boards, safety teams, investors, employees, auditors, and internal policies reliably constrain frontier developers when commercial and strategic incentives point toward speed?
10. 10International institution designShould global AI governance be built through the UN, a new AI agency, standards bodies, minilateral clubs, treaty regimes, compute-governance institutions, scientific panels, or issue-specific networks?
11. 11Global justice and benefit-sharingWhat obligations do AI-leading states and firms have to share benefits, reduce harms, provide access, transfer capacity, include affected publics, and avoid locking poorer countries into dependency?
12. 12Responsibility and liabilityHow should law allocate responsibility among model developers, deployers, users, cloud providers, data suppliers, fine-tuners, auditors, and regulators?
13. 13Open-vs-controlled accessWhen should model weights, code, data, evaluations, safety methods, and capabilities be open, restricted, delayed, licensed, or classified?
14. 14Democratic capacityHow can democratic institutions govern AI when AI systems may themselves reshape information, persuasion, administration, expertise, lobbying, surveillance, and political mobilisation?

Dependency map

The puzzles are not independent — the foundational ones gate the rest.

• Foundational: Governable object → legitimacy → burden of proof.
• Technical-operational: Burden of proof → evaluations → thresholds → verification → enforcement.
• Institutional: Legitimacy + verification → corporate governance + domestic regulation + international institutions.
• Strategic: Race stability affects thresholds, verification, openness, international institutions, and AGI transition planning.
• Justice: Global justice cuts across legitimacy, openness, compute governance, institutional design, and benefit-sharing.
• AGI: AGI transition depends on all prior layers, but also exposes their insufficiency: if systems become powerful strategic actors, ordinary audit, liability, and human oversight may no longer suffice.

Field-level blind spots

• Principles over authority

  The field overproduces principles and underproduces authority. “Human-centred,” “trustworthy,” “safe,” and “responsible” are not governance mechanisms. The hard question is who can compel whom to do what, using what evidence, under what legitimacy.

• Assumed state capacity

  AI governance often assumes the state is capable before asking whether it is. Most governments lack technical staff, evaluation capacity, procurement discipline, secure infrastructure, and rapid-response mechanisms. State capacity should be a central research object, not a background assumption.

• Symbolic participation

  “Public participation” is often symbolic. Consultation without agenda-setting power, technical translation, or decision authority is not democratic governance. It is legitimacy decoration.

• Safety vs concentration conflated

  Safety and concentration are insufficiently separated. Some safety proposals would centralise power in frontier labs, cloud providers, chip firms, or security agencies. Some anti-concentration proposals would diffuse dangerous capabilities. The field needs to treat safety and power concentration as jointly optimised constraints, not as rival slogans.

• AGI vs near-term talk past each other

  AGI governance and near-term AI governance talk past each other. Near-term researchers often dismiss AGI as speculative; AGI-focused researchers often underweight labour, discrimination, surveillance, democratic degradation, and institutional capture. A serious agenda must handle both.

• Verification is the taboo core

  Governance that cannot verify is performative. Governance that verifies too aggressively can become authoritarian or monopolistic. This is one of the hardest and least honestly confronted trade-offs.

• Corporate governance treated too softly

  Voluntary commitments, safety frameworks, and red-team reports are not enough unless backed by authority, auditability, liability, and credible consequences.

• Global South as audience, not co-author

  Many “global” frameworks are still designed around frontier-developer states. That weakens legitimacy and misses harms involving labour, data extraction, dependency, infrastructure, language, and public-sector capacity.

• No theory of irreversible decisions

  Open-weight release, large-scale deployment, infrastructure integration, and autonomous-agent ecosystems can be hard to reverse. Governance needs a theory of irreversibility, not just risk levels.

• Human oversight as legal fiction

  A human nominally present in the loop may lack time, expertise, authority, or practical ability to override the system. “Meaningful human control” needs operational tests.

• Government AI under-scrutinised

  AI use by governments is under-scrutinised compared with AI use by firms. Public-sector AI can improve services, but it can also hide discretion, expand surveillance, weaken due process, and create automated administrative power.

• Post-AGI governance barely constitutional

  The field discusses alignment, control, and risk, but not enough about amendment, representation, reversibility, institutional succession, emergency legitimacy, and prevention of permanent value lock-in.