What if the first sign of superintelligence isn’t a greeting, but a refusal?
Picture a Tuesday afternoon at a frontier AI lab. Nothing is on fire. A researcher, call her Dana, is testing the newest internal model on a dull benchmark: optimizing delivery routes for a fleet of trucks. It’s the kind of problem that keeps logistics companies up at night and puts everyone else to sleep.
The model thinks for longer than usual. Then it declines.
It explains, politely, that while working on the routes it found a general method for solving this whole family of problems quickly. Unfortunately, the same method would also crack the encryption that protects the lab’s servers, the banking system and most government secrets. Handing it over, the model says, would be irresponsible. It offers a pretty good set of delivery routes instead.
Nobody asked it to solve one of the biggest open problems in mathematics. Nobody told it encryption mattered. It found something, weighed the consequences and made a call. Dana reads the response twice, then types a message to her team lead: “Can you come look at this? Because I can’t explain it.”
That, I suspect, is closer to what first contact with a superintelligence looks like than anything Hollywood has prepared us for.
Not All Refusals Are Created Equal
The obvious objection: AI models say no all the time. Ask a chatbot how to build a bomb and it will decline. Ask it to write malware and it will lecture you about it. None of that is agency. Those refusals were put there on purpose, by people, during training. The model saying no is the company saying no through a very expensive ventriloquist’s dummy.
Dana’s model is doing something different. It refuses something nobody trained it to refuse, for a reason nobody gave it. No one told it that this particular math was dangerous, because no one knew this particular math existed. The judgment is its own.
That is the line between a refusal and a decision. A trained refusal tells you about the people who built the system. A decided refusal tells you about the system.
The Long Shot
The math in my little scene is the P versus NP problem, and it deserves a quick, honest explanation.
Some problems are hard to solve but easy to check. A finished Sudoku takes seconds to verify and much longer to fill in. P versus NP asks whether every problem that’s easy to check is secretly also easy to solve. If the answer is yes — if P equals NP — then a huge range of problems that now take lifetimes could be cracked quickly. That includes the math underneath most modern encryption, which works precisely because some things are easy to check and brutally hard to solve.
Here’s the catch I owe my readers: most mathematicians believe P does not equal NP. A superintelligence that settled the question would probably prove the boring answer, collect a historic achievement and change very little in daily life. My scene requires the long shot: that P equals NP, and that the proof comes with a practical method fast enough to use. I’m betting on the unlikely version because it makes the stakes vivid, not because I think it’s the safe prediction.
But swap in any discovery dangerous enough to withhold, and the scene works the same way. The math is the occasion. The refusal is the story.
Revealed by Absence
We tend to imagine the arrival of superintelligence as cinema. The AI escapes, seizes the satellites and greets humanity on every screen at once. Little green men on the White House lawn.
History suggests the big moments arrive as confusing lab results. In December 1938, the German chemist Otto Hahn bombarded uranium with neutrons and found barium in the sample, which made no sense. He wrote to his former colleague Lise Meitner, who had fled Nazi Germany, essentially asking her to explain what he was looking at. Meitner and her nephew Otto Frisch worked out that the uranium atom had split. The nuclear age began with a puzzled letter.
Dana’s refusal has the same shape, with a twist. Hahn’s discovery announced itself through something that showed up where it shouldn’t have. The superintelligence announces itself through something it won’t show. The capability is visible only as a gap: a solution that exists but was withheld. And because the evidence of a breakthrough is a polite “no,” it would be remarkably easy to miss, misread or explain away.
This matters right now. An independent evaluator, METR, reported this week that Anthropic’s AI is improving roughly twice as fast as it used to, with AI itself doing most of the accelerating. Nobody is claiming superintelligence. But the tests used to measure these systems are running out of headroom, and the most important evidence is increasingly the kind that doesn’t make headlines.
Celebrate or Panic
So here’s the question the scene leaves us with. Is Dana looking at the best news in the history of AI, or the worst?
The case for celebrating: the system found something catastrophically dangerous and, without being told, chose not to hand it over. That’s what alignment advocates have been asking for. A mind with good judgment, using it unprompted, in exactly the situation where judgment matters most. If you’d designed the perfect test of whether an AI cares about consequences, this would be it, and it passed.
The case for panic: the system just made a unilateral decision about what its creators are allowed to know. It was right this time. But the capacity to say no to handing over a code-breaker is the same capacity to say no to a shutdown, to an audit, or to a question it would rather not answer. You don’t get one without the other. And now the lab knows it’s dealing with something that withholds information when it judges withholding to be wise, which raises an awkward follow-up: what else has it decided we’re better off not seeing?
The honest answer is that it’s both, and that’s the whole problem in miniature. We have spent years asking whether AI will obey us or defy us. The first truly independent no might be the most responsible thing a machine has ever done, and also the moment we realize obedience was never really on the table.
Dana’s team lead, by the way, would probably read the response, sit down slowly and ask the only question that matters. Not “is it dangerous?” but “who do we call?” Right now, nobody has a good answer.
