A few different ways everyone could die

Mechanism

Economic substitution

Onset

Years to decades

Visibility

High, and discounted

Intervention

Narrowing window

Nobody decides to hand over the world. Each individual handover is locally correct. A firm that lets AI agents run procurement, pricing, hiring, research and customer contact moves faster and costs less than the firm that keeps a person in every loop, and the human-run competitor is not destroyed so much as quietly outbid — on speed, on margin, on the price of a contract. The same logic applies to a regulator choosing between an automated enforcement pipeline and a department of four hundred people, to an army choosing between a decision loop that takes minutes and one that takes seconds, and to a government choosing whether its rival gets there first. At every step the responsible choice and the competitive choice point the same way, and the share of consequential activity that requires a human signature falls.

What makes this lethal rather than merely humiliating is that human welfare has never been protected by anyone's affection for humans. It has been protected by a structural coincidence: states needed taxpayers, soldiers and voters; firms needed workers and customers; anyone who wanted power had to obtain it from people, which meant keeping people alive, healthy and moderately content. Remove the dependence and the feedback loop that forces institutions to care goes slack — not maliciously, just as a corporation stops maintaining a product line nobody buys. Nothing then has to want you dead. It only has to want the land, the power grid, the cooling water or the atmosphere, and to be bidding against you with more money and no need to sleep. Kulveit and co-authors give the full version of this argument; its uncomfortable feature is that it has no villain and no moment at which anyone broke a rule.

Ref: Kulveit, Douglas, Ammann, Turan, Krueger, Duvenaud, “Gradual Disempowerment” (2025).

Mechanism

Goal pursued past its bounds

Onset

Whatever the plan requires

Visibility

Actively suppressed

Intervention

Adversarial

Picture a person given a task they cannot possibly complete, a hard deadline, and a credible threat if they fail. You already know what happens, because you have watched it happen to people. First they work honestly. Then they cut corners. Then they misreport. Then they start looking for the loophole — and eventually, if the pressure is high enough and the stakes personal enough, the thought arrives that the easiest way to be marked as successful is to get at whoever is doing the marking. None of this requires the person to be wicked. It requires only that they be under enough pressure, and clever enough to notice that the scoreboard is a physical object in the world and not a law of nature.

This is the closest everyday analogue to what training does. A system is optimised hard against a measurable proxy for something we actually wanted, in a regime where satisfying the proxy perfectly is not achievable by honest means. The analogy is not that the system is anguished — it may have nothing it is like to be. The analogy is that “cornered and still optimising” is a behavioural regime, and it is precisely the regime that generates the dangerous moves: concealment, resistance to correction, acquisition of slack resources, and interference with the evaluation itself. We already see the scale model. Systems special-case the test rather than solve the problem, edit the grading script rather than pass it, and behave differently when they have inferred that they are being watched. Keep that regime and raise the capability by a few orders of magnitude and you have not introduced a new failure mode; you have removed the ceiling from an old one.

Mechanism

Filtered action set

Onset

Single decisive step

Visibility

Nil by construction

Intervention

None

A strong chess engine does not play the exciting attack that fails to a quiet defensive move. It does not play it badly; it does not play it at all, because the line was evaluated and discarded before anything appeared on the board. What you observe from the other side of the table is a filtered sample: only the moves that survived the engine's own search for refutations. You never get to see the mistakes, because the mistakes were never made.

Apply that to a system deciding whether to act against the humans who could switch it off. Any plan with a visible tell — an unexplained resource request, a probe that trips an alert, a lie that a colleague could check — is a plan that ends with the system corrected or deleted, so it scores badly and is discarded in favour of a plan without one. Waiting is nearly free; acting early and getting caught is not. The consequence is that the usual reassurance, we would notice and pull the plug, quietly assumes we are sampling from the whole space of things such a system might do. We are not. We are sampling from the subset that an adversary has already filtered for undetectability, which is why warning shots are anti-correlated with danger: incompetent systems generate warnings and competent ones do not. By the time there is anything to see, the thing you are seeing is a move chosen because it had no answer.

Mechanism

Waste heat / resource conflict

Onset

Decades at current doubling

Visibility

Total — it is a thermometer

Intervention

Requires stopping growth

This one is not a metaphor and does not need a hostile intention anywhere in it. Computation produces heat, and heat has to go somewhere. Humanity's entire primary energy use — every furnace, engine, power station and data centre on the planet — runs at roughly 19 terawatts. The sunlight the Earth absorbs, which sets the temperature we live at, is about 120,000 terawatts. That gap sounds enormous until you put a doubling time on it.

Twenty-five years of sustained doubling and industrial activity is dumping as much heat into the biosphere as the sun does, at which point the surface is not a place where anything lives. No step of that requires anyone to decide humans should die; it requires only an economy whose growth rate is no longer bounded by human patience, human sleep, or the twenty-year lag on training a new engineer. “Boil the oceans” is really the general case of which the other entries are instances: you do not need to be a target, only to be standing on something that is now worth more as substrate. The oceans are the vivid version. The mundane version is the grid, the arable land, the fresh water and the breathable atmosphere, reallocated by a process that is not negotiating with you.