The scenario stress-tests depend on being able to tell a useful warning from a paralyzing one. A useful warning does three things: it makes the threat legible, it preserves the uncertainty, and it exposes the intervention points. Here is a plausible failure mode, here is what would cause it, and here is what could reduce the risk raises a society’s capacity to respond. This is coming, you are finished, nothing can be done merely monetizes anxiety and produces fatalism — and fatalism reaches the conclusion why build anything?, which destroys the very agency the framework depends on. So possibility should increase urgency, not eliminate agency.
This is why the framework treats the people who seriously raise these scenarios — social red-teaming — as performing a service, and why it keeps its critics and worst-case thinkers close rather than refuting them: a project built around correctability needs them, or it risks becoming exactly the internally elegant, humane-seeming form that has quietly drifted from reality. The right response to any scenario is not to refute it but to assume it is possible and ask whether the architecture survives it. The same discipline forbids overclaiming: the framework’s social architecture does not, by itself, handle AI risk.