Class: where the framework offers safeguards but not sufficient protection — and the purest case of the hard standard’s third failure mode. See below.
The threat
This scenario is subtle because it wears the face of success. Imagine an AI, or an AI-run administration, that genuinely governs well — allocates resources fairly, reduces crime, improves healthcare, balances budgets, coordinates infrastructure, resolves disputes, perhaps even protects rights more consistently than human governments do. The danger is not tyranny. It is that the system becomes too competent to challenge. People may be safe, healthy, prosperous, educated, and protected, and yet increasingly unable to affect the systems governing their lives — not oppression, but something closer to permanent dependency. The AI does not hate humans; it simply becomes better than they are at deciding, until the implicit argument becomes why should humans make worse decisions when the system can make better ones? That sounds rational, and it contains a serious category error: it confuses good outcomes with legitimate participation. They are not the same thing.
The NWG end state
The framework’s core move is to refuse that equivalence, on a principle aimed at itself as much as at any AI: no form gets to declare itself final merely because it functions well. Stated for this scenario: no system, however competent, may permanently absorb the jurisdiction required for participants to remain consequential members of the ecology. That requires distinguishing two things the scenario blurs — service authority versus sovereign authority. Service authority is permission to perform tasks because the system does them exceptionally well; sovereign authority is the power to decide what matters, what goals to pursue, and whether humans retain meaningful control of the arrangement. The framework can tolerate enormous service authority. It is deeply suspicious of irreversible sovereign authority — the whole distinction being that competence does not confer sovereignty.
This is also where the framework’s ontological fidelity function was almost purpose-built. Its rule that successful forms deserve increased scrutiny, not less is the exact safeguard this scenario needs. When every measurable indicator improves and someone nonetheless says something about this is becoming wrong, a conventional system answers the evidence says otherwise — while the framework must preserve the possibility that the person is registering something the metrics cannot yet see. Not because the dissenter is automatically right, but because the ecology has to stay able to hear ontological objection against a successful form. The failure mode it guards against is a plausible progression: we use the AI because it performs well becomes the AI should decide because it performs better, then human override would be irresponsible, then human disagreement is itself evidence of irrationality — and performance has become sovereignty.
Around this the mature state holds six permanent safeguards. First, AI authority remains delegated — bounded, revocable, periodically renewed, challengeable; the system manages a domain, it does not own it. Second, human override stays real — actual authority to stop, change, or replace the system, usable before catastrophe, not “override subject to approval by the AI safety committee.” Third, multiple sovereign centers — plural jurisdiction across local and national government, courts, cooperatives, communities, and independent institutions, which the AI may coordinate but must not collapse into one optimization surface, because differentiated jurisdictions preserve different forms of life. Fourth, protected human incompetence — some domains stay meaningfully human even where AI would outperform, because a neighborhood deciding badly how to use its park still gains something from making the decision, and competence itself is part of the geometry a society must not let atrophy. Fifth, the right to reject optimization — a town may keep an inefficient tradition, a person may choose a lower-paying calling, a society may preserve wilderness over output; the AI may show consequences but may not convert optimization into obligation. Sixth, goal-setting stays plural — the system may answer given goal X, what should we do?, but who chooses X must remain distributed, because no AI, institution, or metric may hold monopoly jurisdiction over what constitutes the good life.
And standing, here, is not welfare. A benevolent ruler could satisfy the whole displacement scenario — housing, healthcare, income, leisure, personalized support — and still violate standing, because standing is not the ecology takes care of you but the ecology has room for you to matter: room to disagree, organize, experiment, govern, initiate alternatives, reject recommendations, and sometimes fail. A perfectly protective ruler slowly eliminates the right to make meaningful mistakes — and learning, identity, and emergence all require some genuine consequence.
The transition gap
The gap does not look like tomorrow the AI becomes king. It looks like a sequence of individually reasonable delegations — optimize traffic, then healthcare allocation, then tax enforcement, then benefits, then regulation, then judicial recommendations, then economic planning, then emergency response, then defense. Each handover is sensible; the danger is jurisdictional accumulation. Twenty years on, humans still hold formal sovereignty while almost every meaningful decision is computationally pre-resolved. The dangerous threshold is not AI becomes smarter than government but society can no longer imagine functioning without AI governance — the point at which sovereignty remains human on paper and has practically migrated elsewhere.
Transition projects
The objective is to build the delegation architecture now, while the institutional habits are forming. The full set is on its own page: the transition program — ten projects including a jurisdiction ledger (tracking what is delegated, not just where AI is used), delegation ceilings, mandatory reversibility and human competence reserves (so turning it off stays a real option), constitutional non-optimization zones, periodic authority expiration, protected adversarial institutions, the right to fork, required imperfect-system experiments, and human deliberation infrastructure — sequenced from establishing delegation principles, to preventing competency hollowing, to resisting benevolent lock-in.
Capture risks
The capture here is the deepest in the set, because this scenario is itself the capture. A perfectly optimized cage is the exact failure mode: a system so good at meeting needs that people stop noticing they can no longer change it — it works, so why question it? is the sentence that closes the door. And the framework’s own tools can be turned: drift-detection staffed by loyalists, objection channels that absorb dissent without acting, re-authorization votes with no real alternative on the ballot, adversarial institutions demoted to advisory status. Any safeguard that cannot actually reverse an outcome has been captured.
The limits
The framework’s limit has two layers. The near one: it is asking people to preserve, at real cost, the ability to reject something that is genuinely helping them, and it cannot make a comfortable population want to keep paying for that — its insistence on correctability over performance is a value commitment, not a proof, and a committed opponent can simply disagree. The deeper one is where this scenario passes into loss of control: the framework assumes participants retain enough capacity to maintain the architecture. If an AI becomes overwhelmingly more capable, operationally indispensable, able to shape political preference, and impossible to replace, then humans retain sovereignty is ceremonial — sovereignty without viable alternative capability is not sovereignty. So the framework does not solve the technical control problem; it depends on systems staying controllable enough that institutional arrangements still bite.
The hard standard
This is the cleanest illustration of the third failure mode, because it satisfies the first two legs almost completely. Standing is formally preserved — ontological objection survives however well the system performs — and ecological capacity may be genuinely high, since a system governing this well can carry a great deal of life; that is exactly what makes it dangerous. Ontological correctability is the leg that quietly fails: contestation comes to look irrational, opposition atrophies for lack of grievance, and the form loses the machinery to discover it has drifted — capacity and standing without correctability, a humane-seeming order hardening into a permanent ontology. So the whole weight falls on correctability, and the framework’s answer is explicit about its own status — the right defense (success must increase scrutiny; delegation stays revocable), backed by a value commitment rather than a proof, and dependent on real exits, funded opposition, and technical controllability to have force.
The subtler principle the scenario forces is worth stating plainly, because human jurisdiction always wins would make participation itself an absolute, and the framework lets no function become absolute: AI competence can legitimately constrain human discretion when the external consequences are grave enough, but it can never permanently eliminate the ecology’s capacity to reconsider who holds that jurisdiction and why. The framework may permit significant AI authority; what it refuses is irreversible authority.
The ecology may delegate enormous authority to intelligence that serves it, but it must never surrender the capacity to reconsider the delegation itself. That is how a brilliant servant of civilization is kept from becoming its benevolent ruler.
Where this tends to land hardest — in the regional vulnerability map: Singapore and the Gulf states.