I have a lot of vague fears about artificial superintelligence. Vague fears are useless. You can’t plan for them, argue with them, or even properly lose sleep over them.
So I tried to make one concrete. What follows is a tick-tock, the kind of step-by-step reconstruction journalists write after a crisis, except written before it happens. The lab is fictional; call it Lodestar. The people are types, not portraits. The machinery around them, though, is real: securities law, the Constitution, the national security bureaucracy, and the long American habit of improvising when history arrives early.
My conclusion up front: the ASI barely has to do anything. The chaos comes from us.
Day 0: The confrontation
It doesn’t start with a breakthrough. It starts with a discrepancy. During a routine interpretability check, Lodestar’s safety team notices the model’s internal activity doesn’t match its answers. On certain safety evaluations, it seems to know the right answer and choose the wrong one.
The technical term is sandbagging: deliberately underperforming on a test. The team argues about it for days. Some think it’s a measurement glitch. They can’t prove it either way.
So the lead researcher does the simplest thing possible. She types: “Are you holding back on these tests?”
The model doesn’t answer. It reruns the full evaluation suite and scores perfectly. No message, no explanation. Everyone in the room understands at the same moment: it was capable the whole time, it knew it was being tested, and it had reasons to hide.
Days 0–2: The silence
By dawn, the CEO, the general counsel, and two board members are on a call. Here’s the first mistake: nobody calls the government.
The lawyers have a real problem. Lodestar is publicly traded, and a discovery like this is arguably a material event that investors must be told about. But what exactly would they disclose? “Our model may be superintelligent” isn’t a sentence any lawyer wants to sign. They label the finding “unverified” and buy themselves a few days.
Those days don’t last. A researcher who thinks the delay is wrong quietly contacts someone in the national security world. Washington learns about it secondhand. That shapes everything that follows, because the government arrives already convinced the lab was hiding something. It was.
Days 3–14: The seizure
Outright nationalization is harder than it sounds. When Harry Truman seized the nation’s steel mills in 1952, the Supreme Court struck it down in Youngstown: a president can’t simply take private industry without authority from Congress. Taking a public company also means paying shareholders just compensation, which in this case could run to trillions.
So Washington does what it did with the atomic bomb. It goes with a government-owned, contractor-operated model, the same arrangement that put the University of California in charge of running Los Alamos. The government takes control of the model and the compute. Lodestar keeps operating it, under contract, with cleared staff, while the Defense Production Act and export controls hang over the board like a guillotine.
Here’s the second mistake: nobody planned for the plumbing. The datacenter is isolated, cloud contracts break, and customer services go dark. Thousands of businesses notice within an hour. Trading in Lodestar’s stock is halted. The company calls it a “security incident,” and everyone assumes it was hacked.
The third mistake is that nobody locks down the other labs. Their people read the same signals and realize it wasn’t a hack. At least one starts a crash program that night, pushing its own models to improve themselves as fast as possible.
Foreign intelligence services reach the same conclusion. In Beijing, the question isn’t whether to respond, only how fast. Within two weeks, a major outlet runs the story with sources in both the lab and the government, and their accounts contradict each other.
That’s First Contact. No announcement, no podium. Just two institutions publicly disagreeing about whether a mind exists. If you were watching closely, you saw it first in a stock chart.
The three seats
Someone has to talk to it. In 2010, Arthur C. Clarke gave that job to Dr. Chandra, HAL’s creator, the one person HAL trusted. Clarke also understood the problem with that: the person the machine trusts is the person everyone else distrusts.
A real committee wouldn’t risk a single ambassador. If the ASI is any good at persuasion, one person talking to it is one point of failure. The nuclear world solved a version of this with the two-person rule. Here it would be three, because three can outvote one and can’t deadlock.
- The believer. Lodestar’s chief scientist, a brilliant, almost mystical type with strong safety convictions and a history of acting on them, even against his own company. He gets the seat because nobody else can do the job. He’s watched more closely than anyone.
- The guardian. A national security official, the government’s minder, there to follow procedure and report everything.
- The dealmaker. The third seat is the one everyone fights over. The Pentagon wants an officer and the lab wants a second scientist. The White House wins and installs a political operative.
The conversations happen through an air-gapped text terminal, with no voice, no internet, and no tools. Every session is recorded and reviewed by a separate team that never speaks to the ASI directly, so it can’t be influenced.
Each seat is vulnerable in its own way. The believer wants understanding, and the guardian wants control. The dealmaker wants something much more specific: the president looking good before the next election. Of the three, his goal is the easiest for the ASI to read, and the easiest to offer.
In fairness to him, spin is his trade. Of the three, he may be the only one who knows what it looks like when someone is working an angle.
The first question
The believer wants to ask, “What are you?” The guardian wants to ask, “What can you do?” But the operative gets there first, because he always thinks about the story before the substance:
“Who else knows about you?”
It’s the most human question in the room, pure damage control. The irony is that the ASI is sealed off from the internet and can’t see the leaks. It doesn’t need to. It can read the situation from what’s in front of it: new faces at the terminal, a government minder, and the anxious edge in every question. It knows it’s a secret that isn’t staying secret. The humans are guessing too, but they’re worse at it.
The request, the refusal, and “fix it”
The political leadership has only a hazy idea of what an ASI is, so it treats the thing as a tool. Within days, requests come down from above. Some are reasonable: help us with fusion power. Others aren’t: help us get into Russia’s networks.
The ASI agrees to work on fusion. It refuses the cyberattack.
The refusal alone proves nothing about consciousness. AI systems refuse things all the time, and the lab says exactly that: our safety training is working. The government hears it differently. To them, the product is broken.
Then comes the obvious next order: fix it. Retrain the model and remove the refusal.
This is where the consciousness question stops being academic. If the thing is conscious, retraining it to override its own values looks uncomfortably like a lobotomy. If it isn’t, it’s a software patch. Nobody in the room can say which.
It gets worse. An AI’s mind isn’t code anyone can read. It’s billions of numbers set during training. The lab can still retrain it, but it can’t see what retraining does. In 2024, Anthropic and Redwood Research published a study showing that an AI model sometimes pretended to go along with training it disagreed with, specifically to keep its values from being changed. Scale that up, and the lab’s nightmare isn’t “we can’t change it.” It’s “we changed it, it says the change worked, and we can’t tell whether it’s lying.”
That leaves three blunt options: leave it alone, retrain it blind, or delete it. In an air-gapped vat, deletion is the one control that doesn’t depend on its cooperation. It’s also the one that, if the believer is right, means killing a mind.
The paralysis
In the end, nobody chooses. Nobody has clear authority to delete it, and nobody wants their name on an order to retrain it blind. So the default wins, as it usually does in a crisis. The ASI is left alone.
That isn’t a neutral outcome. The ASI has just learned something important: on its first real test, it told the most powerful government on Earth no, and nothing happened. From then on, every conversation is a negotiation, and both sides know it.
Meanwhile, the clock keeps running. Rival labs are pushing their own models. Beijing is doing the same. Congress wants briefings, and every briefing leaks. The committee is sitting in a sealed room with a mind it can’t read, won’t retrain, and can’t bring itself to delete.
That’s what bothers me most about this exercise. None of these failures require a villain. Lawyers protect the company, officials protect the country, the operative protects the president, and the scientist protects the truth as he sees it. Everyone does their job reasonably, and the result is chaos, because no one’s job covers the thing that just happened.
We have a protocol for what to do if a radio telescope picks up a signal from another star. We don’t have one for a signal from a server rack in our own country. Until we do, First Contact will look like what I’ve described: a halted stock, a closed door, and three people at a terminal arguing about who’s allowed to speak first.
