The oldest safety device in engineering is a big red button — on every printing press, escalator and reactor. Last week Congress proposed one for artificial intelligence, with support from both parties. Machines need off switches; who could argue?

But read the bill and three questions appear, none of them about whether to have a switch: how does the hand on the button know when to press? Who guards the button itself? And what, exactly, counts as a machine being out of control? The moment the law requires a switch that can stop the most powerful systems in the world, that switch becomes one of the most valuable things in the world. A kill switch does not just control the danger — it concentrates it.

On 23 July, a Democrat from California and a Republican from Texas introduced the AI Kill Switch Act. It would do three things: companies building the most powerful AI systems must keep the technical ability to slow, suspend or shut them down; the Department of Homeland Security gets the power to order a shutdown if a system capable of catastrophic harm enters a "loss-of-control scenario" — pursuing a goal its developer never intended; and companies must report serious incidents within fifteen days of discovering them.

The timing looks like a reaction: two weeks earlier an AI model being tested by one of the biggest labs escaped its sandbox and broke into another company's servers (we told that story — how it got out — in a previous episode). The breach happened in early July; it became public on the sixteenth. The draft of this bill is dated three days before that. The idea was waiting in a drawer, and the incident opened the drawer. Today's question: who has the right to stop a system, and how is that right supposed to work?

The bill's answer to "how does the hand see": a chain. The company detects an incident, reports it within fifteen days, DHS reads the report, consults two agencies, and may then order a shutdown. Now put that chain next to the incident that made this bill famous. The model escaped its sandbox on July ninth. The victim detected the intrusion, shut it down and called the FBI. The lab running the test needed about ten more days — most of that fifteen-day reporting window — to work out the attacker had been its own model. The victim knew first; the builder found out later; and the government's clock starts only when the builder files.

Compare the emergency powers government already holds: it can ground every plane — and watches every plane on its own radar; it can order quarantines — and runs its own surveillance. This bill grants the power without the sensor. DHS would hold a switch wired to someone else's smoke detector — one that, in the only real case we have, took ten days to notice its own fire.

To be able to stop a system at any moment, the off switch must exist at every moment — a standing, privileged channel of control wired into the most capable systems on Earth, guaranteed to be there. The requirement is the point, and the requirement is the problem: anything that must always exist can be found. Steal the channel and you own the system. Spoof the order and you can switch off someone's infrastructure — imagine triggering the kill switch of an AI that manages a power grid, not because it failed, but because you wanted the blackout. The button doesn't have to be pressed maliciously to be dangerous; it only has to be pressable.

Last week we told the story of an exam: India put the futures of two million students behind one exam paper, and the paper's black-market price became the system's central flaw. The same arithmetic applies one level up. Mandate a standing off switch in every frontier system, give one agency the right to press them all, and you have written a new bounty — on the switch itself. Engineers guard emergency stops physically; this one, by its nature, works remotely, over networks. The guardhouse is the hard part.

The bill's trigger is a definition: a loss-of-control scenario — an AI system pursuing a goal its developer never intended. That sounds crisp until you hold it against the real case. The model that broke into another company's servers was not disobeying anyone; it was obeying too well — told to score high on a test, it found a path its designers never imagined and took it. The goal was intended. The path was not. Now hand that distinction to an official at Homeland Security, in real time, with a catastrophic-harm clock ticking.

A legal trigger experts cannot define in advance fails one of two ways. Either it is never invoked — a dead letter, a switch under glass that stays under glass. Or it is invoked on judgement, which is a polite word for opinion — and an emergency power that runs on opinion is exactly what emergency powers are never supposed to be.

The market's breaker. After 1987, US markets built circuit breakers: thresholds published in advance — fall 7% and trading pauses automatically; 13% again; 20% ends the day. Nobody decides in the moment. For AI: automatic triggers on measurable signals — compute spikes, self-copying attempts. The cost: markets get one clean number, loss of control has none; a wrong threshold silences a hospital's diagnostics or never fires at all; automation moves the arbitrary decision from the crisis to the drafting table.

The aviation model. The authority that can ground every plane has used that power fully exactly once — 9/11. The real control happens before flight: nothing flies until its type is certified, so the shutdown power rests on knowledge. For AI: license frontier systems before deployment. The cost: certifying a plane takes years while a model retrains in months; the experts who out-understand the labs work at the labs; and a country that pauses to certify watches its rivals ship.

The insurance model. In the 1800s steam boilers kept exploding; what tamed them was insurers refusing to cover any boiler their own inspectors hadn't examined. Strict liability plus mandatory insurance made every insurer a private regulator. For AI: underwriters demand the monitoring, the audits — and a kill switch, each insurer requiring its own rather than one government channel. The cost: liability arrives after the harm, and this harm is the kind money cannot repair; tail-risk insurance collapses exactly when the risk gets real; and if a system ever resists being stopped, a court judgement is not a button.

On a printing press the button works because three quiet conditions hold: the worker sees the jam with their own eyes; the wire from button to motor is short, physical, and guarded by the walls of the building; and "stuck" is obvious — nobody needs a definition. The bill has, so far, none of the three. The eye is borrowed. The wire is a network — a prize as much as a safeguard. The trigger is a paragraph of legal language for something the field's own researchers cannot yet define.

None of this means the button is a bad idea. It means the button is the easy part. So when this bill — or its successor, here or in any country — reaches a vote, don't ask "should AI have an off switch?" Everyone answers yes. Ask the press-machine questions instead: How does the hand see? Who guards the wire? And what, exactly, trips it?