The AGI Transition and Post-AGI Constitutional Problem
Open problem 14
Open problem
What institutions, rules, and technical controls are needed if AI systems begin to exceed human experts across most cognitive tasks, accelerate AI R&D, autonomously act in the world, or become difficult to control?
Why is this problem foundational?
Why it’s foundational
This is the AGI-governance question in its starkest form. Ordinary regulation assumes humans retain meaningful supervisory capacity. AGI scenarios challenge that assumption.
Why it’s difficult
AGI is uncertain in timing, form, and feasibility. Overfitting governance to one scenario risks irrelevance; ignoring the possibility risks being unprepared. The transition may be gradual in metrics but abrupt in institutional consequences.
What approaches have been proposed?
8 competing positions are catalogued.
Competing positions
- Pause until alignment
- Accelerate for defensive advantage
- Controlled deployment
- Public AGI project
- Internationalised AGI development
- Decentralised open development
- Containment and monitoring
- Resilience-first governance
What could make progress
AGI preparedness exercises; model-autonomy evaluations; AI-R&D acceleration studies; emergency legal frameworks; international crisis simulations; formal work on corrigibility and delegation; institutional designs for reversible deployment.
What it would change
It would determine whether AGI governance should prioritise pause mechanisms, emergency powers, compute control, internationalisation, public ownership, containment, or gradual resilience-building.
What sub-questions need research?
The research agenda breaks this into 5 sub-questions.
Sub-agenda
- What observable indicators suggest an AGI transition is underway?
- What powers should governments have before, during, and after such a transition?
- Can deployment be meaningfully reversible once autonomous systems are integrated?
- What institutions should govern AI systems that assist in AI R&D?
- How can post-AGI governance avoid permanent value or power lock-in?
Priority (editor scoring)
Extremely important, but harder to operationalise than near-frontier questions.
- Importance
- 5/5
- Neglected
- 4/5
- Difficulty
- 5/5
- Actionable
- 3/5
- Robust
- 3/5
- Nat’l+int’l
- 5/5
Where does the catalog bear on this problem?
No current catalog instrument resolves this puzzle — which is the point: it is a foundational question the existing rules leave open. Browse the coverage catalog for what the instruments do and don’t say.
Editorial content — a human-authored agenda question, rendered verbatim. No part of this analysis is AI-generated (see the charter).