Open problem 14
The AGI Transition and Post-AGI Constitutional Problem
- AGI
- post-AGI
What institutions, rules, and technical controls are needed if AI systems begin to exceed human experts across most cognitive tasks, accelerate AI R&D, autonomously act in the world, or become difficult to control?
Why it’s foundational
This is the AGI-governance question in its starkest form. Ordinary regulation assumes humans retain meaningful supervisory capacity. AGI scenarios challenge that assumption.
Why it’s difficult
AGI is uncertain in timing, form, and feasibility. Overfitting governance to one scenario risks irrelevance; ignoring the possibility risks being unprepared. The transition may be gradual in metrics but abrupt in institutional consequences.
Hidden assumptions
Some near-term governance work assumes AGI is too speculative to matter. Some AGI governance assumes near-term harms are secondary. Both are poor framings. The hard problem is designing institutions robust to both gradual frontier progress and discontinuous capability jumps.
Competing positions
- Pause until alignment
- Accelerate for defensive advantage
- Controlled deployment
- Public AGI project
- Internationalised AGI development
- Decentralised open development
- Containment and monitoring
- Resilience-first governance
What could make progress
AGI preparedness exercises; model-autonomy evaluations; AI-R&D acceleration studies; emergency legal frameworks; international crisis simulations; formal work on corrigibility and delegation; institutional designs for reversible deployment.
What it would change
It would determine whether AGI governance should prioritise pause mechanisms, emergency powers, compute control, internationalisation, public ownership, containment, or gradual resilience-building.
Sub-agenda
- What observable indicators suggest an AGI transition is underway?
- What powers should governments have before, during, and after such a transition?
- Can deployment be meaningfully reversible once autonomous systems are integrated?
- What institutions should govern AI systems that assist in AI R&D?
- How can post-AGI governance avoid permanent value or power lock-in?
Priority (editor scoring)
Extremely important, but harder to operationalise than near-frontier questions.
- Importance
- 5/5
- Neglected
- 4/5
- Difficulty
- 5/5
- Actionable
- 3/5
- Robust
- 3/5
- Nat’l+int’l
- 5/5
Where the catalog bears on this
No current catalog instrument resolves this puzzle — which is the point: it is a foundational question the existing rules leave open. Browse the coverage catalog for what the instruments do and don’t say.
Editorial content — a human-authored agenda question, rendered verbatim. No part of this analysis is AI-generated (see the charter).