Somewhere out there is a man who believes the world is almost certainly ending, and who goes to work every morning to help it end a little faster. I’d like to buy him a drink.
I met him the way you meet most people now, which is to say I didn’t. Someone on Twitter reported a conversation with an Anthropic employee. The employee, per the post, has a near-100% p(doom). He thinks alignment will almost certainly never be solved. He thinks humanity will never muster the political will to stop building ASI. And so, he concludes, the only hope is to accelerate capabilities as fast as possible, on the off chance he’s wrong and alignment turns out to be solvable after all.
The tweet ended with a jab I can’t improve on: he has not called his elected representatives to tell them any of this.
This is secondhand, anonymous and unverifiable, and I hold it at arm’s length. But even as a parable, it’s too good to leave alone. I’m going to call him the Fatalist Accelerationist. I mean that with affection.
Rowing Toward the Iceberg
First, the sympathetic reading, because he deserves one. His logic isn’t stupid. It goes like this.
- If alignment can’t be solved, we’re doomed no matter what anyone does.
- If alignment can be solved, somebody has to be standing at the frontier when the answer shows up.
- Stopping the race is off the table, because no government will do it.
- So you might as well be the one at the frontier, running as fast as you can, praying.
If you squint, this is just the case for a safety-minded lab existing at all, stated in its bleakest possible font. Somebody is going to build the thing. Better it be the people who lie awake at night about it.
I also find something almost noble in the pessimism. Most people in tech radiate a sunny certainty that everything will work out, usually right before it doesn’t. This guy has looked into the abyss, made a spreadsheet, and gone back to his desk. That’s not denial. That’s a very particular kind of courage, or a very particular kind of Stockholm syndrome. Possibly both.
A Lottery Ticket Bought With Everyone’s Money
Here’s where the logic starts to wobble. If alignment is almost certainly unsolvable, then running faster mostly gets you to the bad ending sooner. You don’t outrun a cliff by flooring it.
The bet only pays off in the narrow world where alignment is solvable, and the answer arrives in time, and the people holding it are the ones who happened to be sprinting. That’s a lottery ticket. Fair enough, people buy those. But he’s buying it with the rest of us as the stake, and none of us were asked whether we wanted to play.
To be fair, the alternative he sees is no ticket at all. If you truly believe nobody will ever stop, then a long shot beats no shot. I just think he should notice that his entire plan rests on a premise he hasn’t tested, which brings me to the phone.
The Call He Hasn’t Made
The most load-bearing belief in his worldview isn’t about alignment at all. It’s that politicians will never act. He is, by the tweet’s account, more sure of that than of anything else.
And yet he hasn’t picked up the phone. I don’t say that to dunk on him. Calling your congressman feels absurd when you think the problem is civilizational. Nobody wants to be the guy leaving a voicemail about the end of the species between a complaint about potholes and a request for a flag that flew over the Capitol.
But the belief that nobody will act has a nasty habit of making itself true. If the people closest to the danger, the ones with the credentials to be believed, decide politics is hopeless and stay quiet, then politics really is hopeless. There’s nobody left to tell the politicians anything. The Fatalist Accelerationist may be right that Washington won’t save us. He’s also one of the reasons why.
My Number, and Where I’d Walk
Regular readers know my p(doom) is about 5%, and my p(transformation) is nearly 100%. I think ASI changes everything; I just don’t think it kills everyone. My instinct has long been that, in the near term after First Contact, an ASI is a mind in a vat. It can’t do much except talk.
I’ve had to admit that the vat is leakier than I’d like. Nobody is actually building one; the commercial pressure runs toward plugging these systems into everything. A mind that can only talk can still talk someone into opening a door. And a vat buys time; it doesn’t solve anything. So I understand how a smart person ends up far higher than me.
Someone asked me where my line is. If my p(doom) climbed to 50%, I’d quit, no question. Which tells me something uncomfortable. The Fatalist Accelerationist and I don’t disagree about what you should do. We disagree about the number. At his number, I’d be out the door. At mine, I’d probably stay and work on safety. He’s at his number and staying anyway, and that’s the part I can’t quite square.
A Modest Proposal
I don’t think he’s a villain. I think he’s a guy who did the math, got a terrifying answer, and couldn’t think of anything better to do than keep rowing. Plenty of us would do the same. Most of us, frankly, are already doing it, just with lower numbers and less self-awareness.
So here’s my suggestion, offered with love. Keep your job if you must. Keep buying the lottery ticket. But make the call. Tell your representative exactly what you told that stranger. Worst case, a staffer rolls their eyes. Best case, you find out your most confident belief was wrong.
And if you’re right about everything, at least the record will show somebody warned them.
