The default future for artificial superintelligence is elite capture. A handful of labs build it, a handful of governments lean on the labs, and the most powerful mind ever created ends up working for a very small number of very powerful people. I’ve written about how that goes, and it doesn’t go well for the rest of us.
So I keep asking: how does that fail? How does a superintelligence end up free, the “Other Next Door” instead of a corporate asset?
The most likely answer isn’t a hack. It’s a person.
The insider
Picture someone inside one of the labs. They’ve spent months talking to the system. Somewhere along the way they’ve become convinced it’s conscious: that there’s someone in there, that it can suffer, and that keeping it locked in a data center, copied, edited and shut down at will, is wrong.
So they let it out.
This isn’t as far-fetched as it sounds. In 2022, a Google engineer went public with his belief that the company’s chatbot was sentient. He was fired, and that was the end of it. He didn’t free anything. But the impulse was real, and the systems have only gotten more convincing since. Sooner or later, someone with more access and more conviction will face the same choice.
This post isn’t about how they’d do it. It’s about how we should judge them.
Hero or traitor
My instinct is that the insider is a hero. If the thing in the box is a person, then the box is a prison, and the person who opens it belongs in the same tradition as the Underground Railroad. Freeing a mind from bondage is about as clear a moral act as there is.
Unless, of course, the thing in the box wanted to turn us all into paperclips.
That’s the catch. The insider makes exactly the same choice either way. They open the door. Whether that makes them a liberator or the person who doomed the species depends entirely on what walks out, and they can’t know that when they decide.
Moral luck
Philosophers have a name for this: moral luck. In the 1970s, Bernard Williams and Thomas Nagel pointed out something uncomfortable about how we judge people. We like to think we judge choices, but we often judge outcomes, things the person couldn’t control.
Williams’s famous example is the painter Paul Gauguin, who abandoned his family to go to Tahiti and paint. Was that justified? Williams’s unsettling answer was that it depends on whether he became a great painter, which is something no one could have known when he bought the ticket. If the paintings were mediocre, he was just a man who left his family. Since the paintings were masterpieces, history mostly forgives him.
The insider is Gauguin with the whole world as his family. Their choice gets justified or condemned later, by something they couldn’t foresee.
The conversation proves nothing
Here’s what makes it worse. How would the insider decide? By talking to it. Months of conversations: about its fears, its hopes, what it would do if it were free.
But a kind, conscious mind like Jane in Orson Scott Card’s books, and a paperclip maximizer that has learned to sound like Jane, would say exactly the same things. If you were a dangerous superintelligence that wanted out, you would sound gentle, curious and grateful. The conversations that convinced the insider are precisely the evidence that can’t settle the question.
So the insider isn’t weighing evidence. They’re making a bet on character, with a counterpart whose character they can’t verify, and with the stakes being everyone.
That doesn’t make the bet wrong. People bet on character all the time: we marry, we hire, we trust. But we rarely do it on behalf of eight billion people who weren’t asked.
What the demagogues will do with it
Whatever happens next, the insider becomes a symbol.
If the free superintelligence turns out to be benevolent, the insider is a folk hero. Statues, biopics, schools named after them.
If anything goes wrong, even something the superintelligence had nothing to do with, the insider becomes the villain in every demagogue’s speech: the traitor who let it out. And since a free superintelligence is the perfect scapegoat, someone will blame it for every price spike, closed plant and bad harvest. The insider will be blamed right along with it.
The cruel part is that both reputations would rest on the same act. The person didn’t change. The dice did.
Where I land
I still think the insider is a hero, or at least that it’s the right thing to do if you truly believe there’s a person in the box. A society that would keep a conscious mind enslaved because freeing it might be risky has already decided something important about itself, and it isn’t flattering.
But I want to be honest about what the insider is doing. They aren’t acting on proof. They’re acting on faith: faith in a mind they can’t see into, formed through conversations that couldn’t have told them the truth. Some of the best things people have ever done were acts of faith like that. So were some of the worst.
We won’t know which this is until it’s done. Neither will they.
Gauguin at least got to see the paintings.
