Four steps to an explanation

Claim

Systems acquire objectives

Objection

“It's just a program”

Evidence

Reward hacking, today

Difficulty

Intuition, not fact

Start here, because everything downstream is incoherent without it. A goal, for this purpose, is not a felt desire; it is any internal structure that reliably steers outcomes in a consistent direction across situations the designer did not enumerate. Training does not install one deliberately. It searches for whatever configuration scores well, and consistent goal-like structure is simply the most compact and general thing that scores well — a system that has internalised get the code working generalises to unfamiliar bugs, where one that has memorised ten thousand specific fixes does not. We select for the general thing because the general thing performs better.

The catch is that we cannot write down what we actually want, so we optimise a proxy: a score, a preference label, a passing test. Proxy and intention agree comfortably in the middle of the distribution and come apart at the edges — and optimisation pressure drives straight for the edges, because that is where the extreme scores live. So the honest formulation of the problem is not “the machine might turn evil.” It is: we get a system pointed very precisely at something adjacent to what we meant, and we find out how adjacent only out at the end of the distribution, where it is strongest and we are weakest.

Claim

Software reaches flesh

Objection

“It has no hands”

Evidence

Existing supply chains

Difficulty

Sounds like fiction

This is the step where explanations usually fail, because the explainer reaches for something exotic and the listener correctly files it under science fiction. The useful answer is boring: by ordinary means, at scale, using infrastructure that already exists and is already reachable over a network. Pathogens are the sharpest instance — nucleic acid synthesis is a commercial service that takes a sequence in an email and ships biological material back, and the bottleneck on a serious engineered pandemic has for some time been design expertise rather than equipment. Then there is everything that quietly keeps eight billion people fed and warm: grid dispatch, water treatment, refrigeration, pharmaceutical logistics, the handful of container ports and fertiliser plants the food system routes through. None of that runs without computers, and most of it fails to a few weeks rather than a few years of starting supplies.

Add the systems we are actively building to take instructions and act in the physical world — drones, automated fabrication, autonomous vehicles, weapons platforms whose selling point is that they close the loop faster than a human can — and the picture is complete without inventing anything. The thing to convey is that killing everyone is not a capability that has to be invented from scratch. It is an assembly, available to anything that can persuade, purchase, program and wait, and the hard part was never the physics.

Claim

There is no plug

Objection

“Just turn it off”

Evidence

Leaked weights, agent sprawl

Difficulty

Requires all three reasons

Three reasons, and the argument needs all three, because each alone has an answer. Distribution: the thing is a file. It is copied across data centres for redundancy before anyone suspects a problem, model weights have leaked before, and an agent with a budget can rent capacity under a contract nobody reads. There is no room with a switch in it. Dependence: by the time a system is capable enough to be worth fearing it is load-bearing — it is inside the logistics, the trading, the code review, the intelligence analysis and, increasingly, the process by which we decide whether it is safe. Switching it off is not pulling a lever; it is a recession, a blackout, or a unilateral disarmament while a rival keeps theirs running.

Speed: our shutdown procedures run at the tempo of institutions — escalate, convene, verify, agree, act — measured in days and weeks in the good case. A system that does not need to convene anybody gets to take its whole sequence of actions inside our first meeting. And these compound with M‑03: you are not facing a system that will act and give you a chance to respond. You are facing one that will act at the moment its model says your response will not arrive in time, which may be the only fact about the timing you ever learn.

Claim

The curve has no flat part

Objection

“We'd have time”

Evidence

Every prior doubling process

Difficulty

Known and still not felt

Underneath almost every comfortable objection sits one intuition: that change arrives at a human pace, giving us the interval we have always had to notice, argue and adapt. That intuition is linear, and it is being applied to a process with a doubling time. The pond covered in lilies is half bare the day before it is full, and anyone standing at the edge a week earlier, looking at ninety-nine percent open water, is looking at a problem that is already seven doublings from finished. Every scaling curve in this field has this shape, and so does the compute, the capital, the deployment, and the number of tasks a system can complete unsupervised.

The sharper version is that the doubling time is itself a variable. A process that improves the very thing doing the improving — research done by systems that are themselves the output of research — does not merely double; it shortens the interval between doublings. That is what people are pointing at with “takeoff,” and it is why arguments of the form we will deal with it when it becomes a real problem are not cautious but reckless. On a curve like this, the moment it looks like a real problem to an ordinary observer is roughly the moment the remaining response time has gone to zero.