Open problem 2
The Legitimacy Problem
- current AI
- frontier AI
- AGI
- post-AGI
Who should decide whether powerful AI systems may be trained, deployed, paused, nationalised, restricted, open-sourced, or integrated into public infrastructure?
Why it’s foundational
AGI governance is ultimately a constitutional problem, not merely a technical-risk problem. The decision to deploy systems that may reshape labour, security, knowledge, and state capacity cannot be left by default to whichever firms or states happen to control compute and talent.
Why it’s difficult
Democratic legitimacy, technical expertise, secrecy, national security, commercial confidentiality, and global affectedness pull in different directions. Public deliberation is slow and information-poor; frontier development is fast, technical, and often proprietary.
Hidden assumptions
Many proposals assume either that governments are legitimate by default, or that expert-led safety institutions can stand in for democratic authorisation. Both assumptions are weak. States may pursue strategic advantage; companies may pursue profit; experts may lack public legitimacy.
Competing positions
- Democratic control
- Expert technocracy
- Market-led innovation
- National-security sovereignty
- Cosmopolitan/global trusteeship
- Industry self-governance
- Public-interest licensing
What could make progress
Constitutional analysis of AI emergency powers; citizens’ assemblies with technical briefing; deliberative polling on deployment thresholds; models of public-interest fiduciary duties; institutional experiments with public-interest seats, vetoes, or supervisory boards.
What it would change
This would determine whether frontier AI governance should be built around agencies, courts, legislatures, international bodies, public trustees, licensing boards, or democratic deliberative institutions.
Sub-agenda
- What AI decisions require democratic authorisation rather than expert assessment?
- Can secrecy-compatible democratic oversight exist for frontier AI?
- Who represents non-users, future generations, and foreign publics affected by AGI?
- What emergency powers are legitimate for pausing or seizing dangerous systems?
- Should frontier labs have public-interest fiduciary duties?
Priority (editor scoring)
The field under-theorises authority while making quasi-constitutional decisions.
- Importance
- 5/5
- Neglected
- 5/5
- Difficulty
- 5/5
- Actionable
- 4/5
- Robust
- 5/5
- Nat’l+int’l
- 5/5
Where the catalog bears on this
Instruments this puzzle names that are in the catalog:
Editorial content — a human-authored agenda question, rendered verbatim. No part of this analysis is AI-generated (see the charter).