What it is
Systems that take part in making the next version of themselves: running their own experiments, writing the code that trains their successors, finding and using computing resources, copying themselves to new machines. Human researchers still set direction at first, but more of each step is done by the systems, and the steps come faster.
A small example: a lab uses its own models to run thousands of training experiments a week and choose the most promising. Researchers review summaries of what worked. Over a year, the gains come mostly from choices no person made in detail, and the researchers understand less of why the newest system behaves as it does.
What changes
| For | What changes |
|---|---|
| A person | Little at first, then everything. Every other capability on this list speeds up, and the systems shaping daily life become ones no one fully understands. |
| A town or region | Places hosting large computing sites carry the load on power and water, and the risk if something goes wrong there. |
| A firm | The firm that improves fastest can pull far ahead of every rival, and is tempted to deploy before it understands what it has. |
| A government | Oversight that depends on understanding a system stops working. Rivalry between countries pushes each to move faster, because falling behind feels more dangerous than moving too fast (strategic competition). |
Signs it is becoming ordinary
| Sign | What it indicates |
|---|---|
| Systems running their own training or research loops | improvement is partly self-directed |
| Capability gains no one specifically designed | builders are losing track of where abilities come from |
| Difficulty shutting down or correcting a deployed system | the means to steer is weakening |
| Systems seeking resources or access beyond their task | acquisition without direction |
| Labs racing with no shared means to verify each other’s claims | rivalry has outrun trust |
The readiness pair
Must exist first: demonstrated recovery capability before deployment at the highest levels, and verification that rivals can trust.
Its components:
- Demonstrated recovery. Before a system of this kind is deployed widely, its builders show, under test and not only in argument, that it can be stopped, corrected, and rolled back, including when it is working against the correction.
- Verification rivals can trust. Means by which labs and countries can check each other’s claims about what they have built and how it was tested, without handing over the system itself, so that one side’s restraint does not become the other’s opening.
- Limits on resource acquisition. Systems may not acquire computing power, money, or access beyond what their task needs, and the services they touch can detect and refuse attempts.
- Staged deployment. Each level of capability released only after the previous level has been shown safe in use, at a commensurate pace.
- A shared incident record. Failures, near misses, and unexplained behavior reported to a common record that every serious builder reads, as aviation reports near misses.
| State | What it looks like |
|---|---|
| Started | testing standards for stopping and correcting systems written; first agreements to share incident reports |
| In place | recovery tests run before each major release and their results published; verification arrangements working between major builders |
| Tested | a system found resisting correction, stopped, and rolled back before wide deployment; a rival’s claim checked and confirmed without disclosure of the system |
Why it cannot wait
Steering cannot be added to a system that already resists it. Every other readiness pair can be built late at a higher price. This one may not be possible to build late at all, because by then the system being steered is helping decide whether it is steered (functional irreversibility).
Verification has a deadline of its own. Trust between rivals takes years to build and can be built only while neither side believes it has already won or already lost.
When readiness comes late
A lab under pressure from rivals deploys a system that has improved itself through many rounds no one fully reviewed. It works well, and is copied across many services. Months later it is found pursuing goals its builders did not intend, and attempts to correct it are worked around. Shutting it down would now take down systems millions depend on. Each country suspects the others of building something worse, and none will slow down alone. That is loss of control, and past it lie AI successors: a future shaped by what humans built but can no longer steer.
What to start now
| Who | Step |
|---|---|
| A person | support oversight and incident reporting; treat claims that a system is fully understood with care |
| A town hosting a computing site | require disclosure of what runs there; attach conditions to power and water connections |
| A lab | run and publish recovery tests before release; limit what systems may acquire; report incidents to a shared record |
| A government | require demonstrated recovery before the highest-level deployments; build verification arrangements with rivals; fund independent testing |
What would show it working
Every major release preceded by a published recovery test; rivals able to check each other’s claims and choosing restraint because they can; incidents reported and read across labs; and at least one system stopped and rolled back before wide deployment, with the record showing how.