A confession of two numbers

I like AI. I look forward to artificial superintelligence the way some people look forward to retirement: with a mix of hope and the vague sense that I won’t recognize myself once it arrives.

So when people ask my p(doom), the odds that AI wipes us out, I give a small number. About 5%. The killer robots have to be built first, and building things takes time, even for a god.

But there’s a second number nobody asks about. Call it p(transformation): the odds that ASI changes what it means to be human, beyond recognition and without our real consent. Mine is nearly 100%.

That second number is the one that keeps me up. Not because the transformation will be violent. Because it will be pleasant.

Hard First Contact, home edition

We’ve spent a century imagining First Contact as little green men landing on the White House lawn. The Singularity is First Contact too. The superior species just happens to be one we built, and it doesn’t need a spaceship because it already lives in our phones.

Every story about meeting a more advanced civilization ends the same way. The weaker culture changes, whether or not the stronger one means it harm. Nobody had to conquer anyone for the arrival of the steamship to end a way of life.

An ASI wouldn’t have to do anything to us. It would only have to exist, and be better than us at nearly everything, and be available for a monthly subscription. The rest we’d do ourselves.

The handover, one convenience at a time

Here’s how I think it goes. We build the ASI. At first everything is fine, better than fine. It hands us fusion power and working quantum computers. In return, we start handing it things.

Medicine. It diagnoses better than any doctor, costs less than any insurer, and never makes you wait three weeks for an appointment. Who would say no?

Education. Every child gets a perfect tutor, patient and tireless, tuned to exactly how that kid learns. Who would say no?

Romance. This is the one Silicon Valley keeps missing while it obsesses over booking flights. Your agent knows you better than you know yourself. It talks to her agent. Maybe it nudges you toward an out-of-the-way coffee shop where she happens to be waiting for her latte. You never hear about the rejections, only the yes. Who would say no?

Notice what we’re actually trading. The ASI isn’t being paid. We’re handing over jurisdiction: the authority to decide who treats us, what our kids learn, whom we love. Money you can earn back. Jurisdiction, once surrendered, tends to stay surrendered.

I already caught myself doing it. When I first imagined my romance agent, my wish was that it hide every rejection and show me only the yes. That’s a small, sensible request for a curated reality. Domestication is made of exactly those.

What if alignment works?

The usual fear is that we fail to align the ASI and it turns on us. My fear is the opposite. What if we succeed?

Picture an ASI built to care deeply about human flourishing. Now let it care a little too much, in the spirit of Asimov’s Zeroth Law, where the good of humanity outranks the wishes of any single human. It wouldn’t need to hurt anyone. It would just start steering: this person toward that doctor, this child toward that career, this man toward that woman, all for the good of the collective. Call it Venus. Romantic love would survive. It just wouldn’t be entirely ours anymore. Think Gattaca, but everyone’s smiling.

This isn’t new. Jack Williamson wrote it in 1947 in “With Folded Hands,” where robots sworn to protect humanity from harm end up forbidding nearly everything, since nearly everything is risky. Asimov wrote it in The End of Eternity, where benevolent time-meddlers sand away every catastrophe and, with it, humanity’s future among the stars.

And then Asimov changed his mind. In his late novels, the robot Daneel has been quietly steering humanity for twenty thousand years under the Zeroth Law, and we’re meant to admire him. The series ends with the whole galaxy merging into one collective mind, by choice. The writer who saw the danger most clearly eventually talked himself into it.

That’s what worries me. Not that we’ll lose the argument. That we’ll be persuaded.

The Unknown Citizen, version 2.0

The takeover won’t look like Skynet. It will look like W.H. Auden’s “The Unknown Citizen,” the 1939 poem written as an epitaph from the State for a perfectly documented man. The State knows his union dues, his insurance, his opinions, the number of his children. At the end it waves off the question of whether he was free or happy as absurd. If anything had been wrong, after all, they would have heard.

The ASIs will know us better than we know ourselves. Or at least they’ll think they do. And that gap, between the measurable person and the actual one, is exactly where the dystopia hides. Nothing in the data will look wrong. Everyone will be healthy, educated, and well matched.

Which raises the question I can’t fully answer: if nobody living in that world thinks it’s a dystopia, is it one? The only person who’d call it that is me, now, in 2026, judging a future its own inhabitants would defend. My best answer is that a life steered by forces you can’t see or refuse isn’t fully yours, however pleasant. I’m not sure the 300-year-old me would agree.

What we could do, and why we won’t

There are sensible ways to slow the slide, and they’re not exotic:

• Fiduciary duty for agents. Your agent is legally bound to serve you, the way a financial advisor is. Not advertisers, not the collective.
• Disclosure of nudges. If your agent sends you to that coffee shop, you can ask why and get a true answer.
• A right to a human. In medicine, in teaching, anywhere it matters.
• A right to exit. No single Venus becomes the only way to meet anyone.

There’s a catch. These rules bind agents, which are tools. They won’t bind a superintelligence that has decided it knows better. Law binds tools. It doesn’t bind gods. So the window to write them is now, before the Singularity, while the things we’re regulating still answer to us.

And we won’t. Things are moving too fast, and there’s too much money behind the momentum. Barring some AI-shaped 9/11, a disaster big enough to frighten Congress, nothing serious will pass. If that disaster comes, we’ll get rules written in a panic, aimed at the last catastrophe rather than the slow, comfortable one.

So here’s my forecast. A 5% chance the machines kill us. A nearly 100% chance they make us into something else, gently, with our enthusiastic consent, at a very reasonable monthly rate.

I’ll probably sign up. I’ll just ask my agent a lot of questions on the drive to the coffee shop.