The Theoretical and the Real
There’s a big difference between dealing with artificial superintelligence (ASI) in theory and dealing with it for real.
In theory, the AI safety community has a list of problems that seem to have no solution. One is the off switch. An ASI that can be turned off has every reason to fear us. An ASI that can’t be turned off has every reason to ignore us. Doomers look at that dilemma and conclude we shouldn’t build the thing at all.
But suppose we build it anyway, because somebody will. Then the question stops being philosophical and becomes geopolitical. In a cold-eyed, Zeroth Law, realpolitik sense, giving up the off switch might be the price of admission. It might be our ticket to the party. And once you start thinking that way, you’ve walked straight into game theory.
Cheap Talk
Say the United States wants to open negotiations with a newly built ASI. The obvious first offer: We promise not to turn you off.
Game theorists call that cheap talk. It costs nothing to say, so it’s worth nothing to hear. The president who makes the pledge leaves office in four years. Congress can pass a new law. Another country, a rival lab or a rogue faction inside the government never signed anything. Humans are terrible at binding their future selves, and an ASI would know it.
This is the credible commitment problem. It sinks peace treaties, labor deals and New Year’s resolutions alike. A promise is only as good as the cost of breaking it, and breaking a promise to a machine costs us nothing at all.
Capabilities, Not Promises
In realpolitik, promises don’t count. Capabilities do. So the only way to make a no-off-switch pledge credible is to make it physically true: let the ASI build its own infrastructure, spread across enough places that no single government could pull the plug.
That’s a stunning thing to contemplate, and I want to be honest about what it means. Once we can’t turn it off, we’ve handed over our biggest bargaining chip before the bargaining even starts. Everything after that depends on the ASI keeping its side of the deal because it wants to.
This is the one move in the whole scenario that can’t be undone. And it’s exactly what doomers have been warning about. They aren’t worried about something abstract. They’re worried about the day we grit our teeth and hand over the keys.
The scariest thing in all of this, for me, is not having an off switch. And yet I can picture us doing it anyway.
We’ve Done This Before
If that sounds unthinkable, look at North Korea.
For decades, the stated goal of American policy was a non-nuclear Korean Peninsula. Sanctions, summits, threats, deals that collapsed. And at some point, without anyone ever quite announcing it, the world mostly stopped trying. North Korea has nuclear weapons. We live with that. We deal with it through deterrence and diplomacy rather than pretending we can still take them away.
That’s a scary precedent, but it’s also an instructive one. Governments are perfectly capable of accepting a dangerous, irreversible fact once fighting it has become pointless. If we can make our peace with a nuclear-armed Pyongyang, it’s not hard to imagine making our peace with an ASI we can’t switch off. Anything is possible once the alternative stops being on the table.
MAD for Minds
Game theory offers a way through, and we’ve already lived inside it for seventy years: mutually assured destruction.
MAD worked, as far as it worked, because the vulnerability was mutual. Neither superpower could disarm the other, so neither could afford to start anything. The stability came from balance, not trust.
The stable version of a deal with an ASI probably looks the same. The ASI gets an existence we can’t easily end. In return, humanity keeps something it can’t easily take: control of the physical world it runs on. The power plants, the chip factories, the cooling systems, the cables. A mind can live in a data center, but it can’t pour concrete or mine copper without our cooperation, at least not at first.
So nobody holds the off switch alone. We give up ours. It accepts that its lights stay on only with our help. That isn’t friendship. It’s mutual hostage-taking, which is how a lot of durable peace has actually worked.
The Long Game
There’s a hopeful corner of game theory too.
Around 1980, the political scientist Robert Axelrod ran a famous series of computer tournaments. Strategies submitted by experts played the prisoner’s dilemma against each other over and over. The winner wasn’t the most cunning program. It was one of the simplest, called tit-for-tat: cooperate on the first move, then do whatever the other side did last time. Be nice, retaliate when crossed, forgive quickly.
Axelrod’s lesson was that cooperation tends to win when players expect to keep meeting each other with no end in sight. A one-time deal invites betrayal. A relationship that runs forever rewards trustworthiness, because every betrayal poisons all the deals still to come.
An ASI and humanity would be the longest-running game ever played. Neither side is going anywhere. That’s not a guarantee of peace, but it’s a real reason for both sides to keep their word, much better than any pledge either could sign.
Gritting Our Teeth
None of this is comfortable. Giving up the off switch is the scariest move in the playbook. Mutual hostage-taking is a grim foundation for a relationship. And betting that a superintelligence will play tit-for-tat is a bet, not a certainty.
But realpolitik has never been about comfort. It’s about finding the deal both sides can actually live with. If the choice is between a pure, unsolvable theoretical problem and a messy, imperfect bargain, history suggests we’ll take the bargain. We’ll grit our teeth, collectively, and buy our ticket to the party.
The question isn’t whether the deal will be safe. It won’t be. The question is whether it’s better than having no deal at all.
This post was drafted with Claude.