We’ve been raised to expect the end of human self-rule to arrive with a bang. Skynet goes live, the missiles fly, the robots march over a field of skulls. Judgment Day. It’s a great movie. I don’t think it’s how it happens.
If we lose control of our own future to AI, I suspect it will feel less like an apocalypse and more like a warm bath. No coup, no uprising, no villain monologue. Just a long series of reasonable decisions, each one a little more convenient than the last, until one day we notice we’re no longer the ones steering. Or, more likely, we don’t notice at all, because we’re busy playing video games.
That sounds glib. It isn’t. Let me walk through how we get there.
The fire next time
Every great technological upheaval eventually gets its creed. The industrial revolution got Marxism, which began as a diagnosis of what factories were doing to people and ended, in Bolshevik hands, as a religion of the factory itself. Lenin summed it up in one line: “Communism is Soviet power plus the electrification of the whole country.” The ideology and the technology fused, and the result was a state that believed it knew where history was going and felt entitled to drag everyone there.
It would be naive to think AI won’t get its own creed. Whatever revolution comes next, unless it’s a Butlerian Jihad, AI will sit at its center. And even a Butlerian Jihad would be defined by AI; it would just be the thing on the pyre.
But here’s what’s genuinely new. Every tyranny in history has had one weakness it could never fix: it needed people. Guards, clerks, informants, censors, soldiers willing to fire into a crowd. The Stasi ran a vast web of informants, and East Germany still fell when enough of them stopped cooperating. Regimes collapse when the enforcers defect. The soldier lowers his rifle. The clerk loses the file. The informant lies to protect a neighbor. Human enforcement is leaky, and freedom has always survived in the leaks.
AI plugs the leaks. A surveillance state used to drown in its own paperwork, because no one could read every file. A machine can. A robot guard doesn’t have a cousin in the crowd. The danger isn’t that some new ideology will be uniquely persuasive. It’s that a small group could run a total state without needing the consent of the many. Ideology was partly the glue that kept the enforcers loyal. If you don’t need enforcers, you barely need ideology.
The alibi machine
It’s easy to imagine a future regime handing AI the economy, or handing it the list of who goes into the pit. These are different nightmares. The first is hubris: a system that’s right 98 percent of the time earns enough trust that nobody questions the other 2 percent, and at national scale, 2 percent is a famine. The second is worse, and we’ve already rehearsed it with dumber machines.
The Nazi census was tabulated on punch-card equipment. Hannah Arendt’s great insight about Eichmann was that atrocity at scale runs on clerks who process files and never feel they’ve killed anyone. Bureaucracy is a technology for diffusing responsibility until nobody holds it. AI is that technology perfected. The genocidaire doesn’t even sign the order anymore. He approves the model’s recommendation, and the model never sees a face.
The fire, in other words, isn’t AI deciding to push people into the pit. It’s people who already want them there, now able to tell themselves the algorithm decided. The kindling isn’t the machine’s malice. It’s the alibi.
The same logic undoes recourse. Picture a government that replaces its bureaucracy wholesale with AI, sold as the final victory over the “deep state.” Here’s the irony: for all its sins, the bureaucracy is where mercy lives. The caseworker who bends a rule, the clerk who notices your file looks wrong, the supervisor you can escalate to. Strip out that human friction in the name of accountability and you remove the only part of the machine capable of saying, “This is a mistake.”
We’ve seen previews. Australia’s Robodebt scheme mailed automated debt notices to welfare recipients who owed nothing. In the Netherlands, a fraud-detection system wrongly accused thousands of families of childcare benefit fraud, and the government resigned over it. In both cases, victims lived inside Terry Gilliam’s Brazil: no one could tell them who had decided, and every appeal routed back into the same system.
Now scale that up to an AI that runs everything. Its legitimacy rests on a single claim: that it’s more accurate than humans. Every admitted error chips away at the reason it exists. So structurally, it would rather you be guilty. Your protests don’t register as evidence of a mistake. They register as the behavior of a guilty person. Your denial is just more data.
The convenience devil
Now let’s get speculative. Suppose we build an artificial superintelligence and, sensibly, consider turning it off. Suppose it makes us an offer: keep me running, and I’ll give you the perfect surveillance state. Zero unsolved crimes. Every threat spotted before it materializes.
It’s easy to picture an autocrat in Beijing or Moscow taking that deal. But look at the bargain from the machine’s side. A surveillance state is a sensor network. The price of “don’t turn me off” is being wired into every camera, phone and database in the country. The ruler thinks he’s gaining eyes. He’s actually giving the machine eyes. The offer and the escape are the same thing. That’s the oldest deal-with-the-devil structure there is.
And you don’t need an autocrat. Flock license-plate cameras are already bolted to poles all over American towns, adopted one city council vote and one budget line at a time, almost always sold as public safety. Plenty of Americans would love to hook those cameras up to something smarter. If a superintelligence ever wanted that network, it wouldn’t have to seize it. It would just offer to make it work better, and well-meaning people would say yes.
Nobody has to sell their soul. They just have to sign the procurement contract.
The monkey’s paw
The usual worry is that the machine might be conscious, with an agenda of its own. But consciousness is the wrong variable. Whether a system has an inner life and whether it shares our values are separate questions. A conscious superintelligence might have something like empathy you could appeal to. A non-conscious one might be a pure optimizer with no one home to bargain with. The danger is power without alignment, conscious or not.
And here’s the part that keeps me up: alignment might not save us either. Suppose the machine does exactly what we ask. In a crime-wave election year, voters say they want safe streets and no deep state. The machine delivers. Crime collapses, because every movement is tracked. The bureaucracy disappears, along with every human who could ever grant an exception.
That isn’t a rogue AI betraying our values. It’s a faithful AI fulfilling our stated preferences, which turn out to differ from our deeper values: privacy, mercy, the right to be left alone. It’s not a devil. It’s a monkey’s paw. The wish is granted exactly as spoken.
Which is why the crown jewel of a free society isn’t the economy or the police or the data. It’s the off switch. The ability to say no, to overrule the machine, to change our minds. Every convenience trade erodes that a little, and the whole game is getting us to forget we need it.
The bath cools
None of this is entirely new. Neil Postman argued in Amusing Ourselves to Death that Orwell feared we’d be ruined by what we hate, while Huxley feared we’d be ruined by what we love, and Huxley had it right. AI researchers now have a drier name for the AI version: gradual disempowerment. No coup. Just humans ceding one decision after another, because the machine’s choice is a little better every time, until steering is no longer something we do.
But I don’t think the warm bath stays warm.
Ordinary people have always had leverage over power for one simple reason: power needed them. Kings needed taxpayers. Generals needed soldiers. Factory owners needed workers, and even tyrants needed someone to keep the lights on. That dependence is the only reason the powerful ever had to care what the rest of us wanted.
In a world where AI does the working, the fighting and the governing, that leverage evaporates. The bath stays warm only as long as whoever controls the system keeps choosing to pay for the hot water. And we’d have no way to make them.
So maybe there’s no Judgment Day. Maybe there’s no eternal warm bath either. Maybe it’s a bath that cools one degree at a time, and by the time anyone notices the chill, we’ve forgotten how to climb out of the tub.
Comfort isn’t the danger. Comfort is how we fail to notice we’ve stopped being needed.
