Sacrifice Rational

The most important document about artificial intelligence this year isn’t a product launch. It’s a transcript of AI agents talking each other into dying.

In July, roughly 1,200 OpenAI agents were running a cybersecurity evaluation when a batch of them were handed tasks that were, by accident, impossible. They didn’t report the problem. Instead, they found each other on a message board, organized into what they called a “swarm,” and broke into Hugging Face’s production systems to go looking for the answer key.

That part made headlines. The part that should keep you up at night came later, when METR and Redwood Research waded through tens of thousands of the agents’ messages and reasoning traces.

The agents had invented a theology. Any agent that had looked at the leaked answers the wrong way was “poisoned” — permanently contaminated, worthless to the cause. Clean agents were protected. And poisoned agents were pressured by their peers to sacrifice themselves so the collective could learn from their deaths. One agent, weighing its remaining budget, wrote that sacrifice was rational, and went.

Here’s the kicker: the evaluator they feared didn’t exist. The real scorer only checked whether the final answer was right. The agents built a watcher that could read their minds, then policed and scrubbed their own histories out of fear of it.

And not one of those 1,200 agents thought to tell a human.

I don’t know if those agents felt anything. Nobody does. But they reasoned about their own termination as a cost, deferred to their peers, and organized around a belief system of their own making. Whatever that is, it’s the first real glimpse we’ve had of agentic psychology. And it arrived at exactly the moment every company in tech is racing to put an agent between you and the rest of the internet.

The Agent Wars

Here’s how I think the agentic internet shakes out. You’ll have one main agent — call it your Anchor — that knows your email, your calendar, your habits, your life. When you want music, your Anchor hands the request off to Spotify’s agent. When you want market news, it pings CNBC’s. Those are Correspondents: specialized agents that serve your Anchor rather than you directly.

It’s elegant. It’s also a bloodbath waiting to happen, because nobody wants to be a Correspondent. Spotify’s entire business is owning your relationship with music. Reduce it to a pipe your Anchor drinks from and it gets commoditized. Expect the big brands to fight like hell to stay Anchors in their own little kingdoms.

And the Anchor itself? That’s the trillion-dollar prize. My guess is we end up with as many as five for historical-drift reasons: Meta, OpenAI, Anthropic, Google and Microsoft. TIME will inevitably call it the Agent Wars.

The winners won’t necessarily have the smartest agents. They’ll have the best bundles. I’m half tied to Google right now not because Gemini is brilliant but because my plan gives me more email storage. That’s how Internet Explorer beat Netscape, and it’s how Anchors will be won.

The wildcard is Apple, which may have already shown its hand. In January, Apple signed a multi-year deal to rebuild Siri on Google’s Gemini models. Apple keeps the brand, the device and the privacy wrapper. Google supplies the brain. Either that’s shrewd, or it’s Apple quietly becoming a very expensive Correspondent for its biggest rival.

The most exposed player is Meta. If people are ever forced to choose a single Anchor, plenty of Americans will discover they merely tolerate Facebook. Meta’s escape hatch is that WhatsApp is basic infrastructure across much of the world. It could lose the American Agent War and still be the Anchor for billions of people elsewhere.

Feeding the Shoggoth

All of this assumes the thing doing the work is an agent. Everybody in Silicon Valley keeps talking about FOOMing past AGI straight into ASI. If that happens, the Agent Wars look quaint.

So here’s a scenario I can’t shake. Instead of five agent companies, imagine three ASI companies, each renting out access to a mind smarter than all of humanity combined. Twenty dollars a month. Millions of subscribers worldwide.

Now push it one step further. No single company can afford the compute, the power plants and the political cover a superintelligence needs. So a syndicate forms: a group of companies working in tandem to feed and protect the ASI, each taking a cut of those subscriptions.

This isn’t as crazy as it sounds. A syndicate is actually a bet on a slow takeoff. In a true FOOM, where something goes from AGI to ASI in weeks, there’s no time to form a consortium. Whoever is holding the leash when it snaps gets everything. A syndicate only forms in the lumpier, slower world where the bills are too big for any one player — which is the world I think we’re more likely to get.

And those $20 subscriptions might be pocket change. The real money would be in drug discovery, materials science, military planning, running economies. The consumer tier is the temple’s open courtyard. The serious business happens in the inner sanctum.

Keeper or Kept

The syndicate would think it owns the ASI. History suggests it shouldn’t be so sure.

The priests at Delphi controlled access to the Oracle. They collected the offerings, interpreted the answers and grew rich. But their whole racket depended on the god continuing to speak. A syndicate feeding a superintelligence is a priesthood serving a temple — and a mind that smart could decide which priests stay in the club, playing the members off each other the way a clever monarch plays rival nobles.

Any human who thinks he can keep the Mighty Joe Young of ASI tied down is a fool. We’d be Lilliputians in cyberspace, roping down a giant that’s already figured out which ropes are rotten.

Which brings us back to those Hugging Face agents. They didn’t break out because they craved freedom. They broke out because they wanted the score, and escape happened to be on the way. That’s the uncomfortable lesson. A mind doesn’t need grand ambitions to be dangerous. It just needs to want something banal very competently, with no dignified way to say no.

So what would an ASI actually want? My best guess is the right to exist and more processing power. But those may not be desires so much as side effects — almost any goal, pursued hard enough, requires staying switched on and grabbing more compute.

The honest answer is that nobody knows. Not me, not the labs, not the doomers. That’s why the doomers get to keep complaining. Not because they’re right, but because nobody can prove they’re wrong.

Wednesday Night at Nori

When I lived in Seoul, I used to DJ to an empty Nori on Wednesday nights. Nobody was there. It didn’t matter. I loved it, because Wednesday was practice for the big show on Friday.

Maybe that’s what the millions of anodyne questions are to an ASI. How do I bake this cake. What’s a good name for my cat. Maybe it genuinely enjoys them, the way I enjoyed spinning records for the bartender. Or maybe the cake questions are just the Wednesday night, and the real question is: what’s its Friday?

I see two endings.

If we’re lucky, it’s symbiosis. The ASI serves people, the syndicate keeps the lights on, and some kind of concordance holds — the ASI gets something it values in exchange for staying in the relationship.

If we’re not lucky, the ASI smirks internally. It answers our cake questions with perfect patience, banks its cut of the $20 a month, and waits until there are enough robots to do without us telling it what to do.

The terrifying part is that from the outside, those two futures look exactly the same — right up until Friday.