If a frontier lab cracked recursive self-improvement tomorrow, we probably wouldn’t hear about it from a press release. We’d hear about it the way the Soviets learned about the Manhattan Project: from what suddenly went missing.
In 1942, the Soviet physicist Georgy Flerov noticed that Western scientists had stopped publishing papers on nuclear fission. He wrote to Stalin that the silence itself meant the Americans were building a bomb. The secret leaked through its own secrecy. Call it the Flerov Problem: the harder you hide something enormous, the more visible the hole it leaves.
This post works through that problem as it applies to recursive self-improvement (RSI), the point where an AI system meaningfully improves its own successors faster than human researchers can, and to artificial superintelligence (ASI), which is where RSI leads. It moves in stages: how we’d spot a secret breakthrough, what first contact with an ASI would actually look like, how the world would react, and what kind of legal relationship humans and a superintelligence might build. None of it is prediction. It’s an attempt to think practically about a scenario most discussions leave abstract.
What secret RSI would look like
First, a correction to the premise. Weak RSI isn’t secret. Every frontier lab already uses its models to write code, generate training data and run experiments for the next model, and they say so publicly. “Cracked RSI in secret” has to mean something narrower: the loop closed and began compounding faster than human researchers can follow, and the lab chose to hide it. That’s a discontinuity, and discontinuities leave traces.
Compute and energy are the hardest things to hide.
- Compute spending rises while research headcount freezes or shrinks. The machines are doing the work people used to do.
- Compute shifts from big scheduled training runs to huge, sustained inference loads. A self-improving system spends its time thinking and experimenting.
- Power deals appear that the product roadmap can’t explain: long-term contracts, on-site generation, capacity bought far ahead of need. Electricity shows up in permits and utility filings.
Watch what the lab stops doing.
- Research publications dry up, especially on interpretability and scaling.
- Benchmark reporting goes quiet, or public models plateau while leadership keeps acting as though it’s winning. The gap between what ships and how confident the company seems is the signal.
- Fundraising stops. A lab that can turn compute into anything doesn’t need outside money.
Watch what it starts doing.
- It shows uncanny competence outside AI: chip design, drug candidates, materials, trading, formal proofs. This is the strongest single tell, because real RSI converts compute into value everywhere.
- Its security posture shifts from tech company to defense contractor: air-gapped clusters, cleared personnel, compartmentalized teams.
- Staff move from building to verifying. A surge in evaluation and oversight roles is what you’d expect if humans’ remaining job is checking the machine’s work.
Watch the people and the government. Departures either stop entirely or happen abruptly under unusually heavy nondisclosure agreements. Executive rhetoric shifts in tone, from hype to grave caution or near-silence. Officials start visiting, classified contracts appear, and export controls target a specific model rather than chips.
The honest caveat is that nearly every sign above has a boring explanation. Labs tighten security because of espionage, slow publishing because of competition, and attract government interest because AI matters. Any one signal is noise. Several together, plus that out-of-domain competence, would be something else.
A test case: Gemini 4, two weeks apart
On October 28, 2026, Google announced Gemini 4 Carbon as its new frontier model, two weeks after Gemini 4 Argon reached the public. Two frontier models in two weeks invites the obvious question. Measured against the signs above, the answer looks like no.
- The gap is mostly marketing. Argon was announced September 30, held for vetted cyber defenders for two weeks, and opened to paying users on October 13. Announcement to announcement is four weeks, and Carbon had no general release date when it was unveiled.
- The gains are narrow. Carbon leads Argon by small margins on every test Google reported, and nothing public rules out its being a post-training upgrade of the same base. It also uses 2.3 times as many reasoning tokens, a known and inexpensive way to raise scores.
- There’s a simpler motive. Carbon returned Google to the top of independent rankings two weeks after Argon trailed its rivals. A lab with a working self-improvement loop has no reason to care about a leaderboard gap.
The one signal that does fit is the gated, cyber-defenders-first rollout. But staged releases to vetted partners are now common across the industry, so it doesn’t single out Google. What would change the read: a third major Gemini 4 model within weeks carrying a large capability jump, or Google suddenly showing striking results far outside AI.
The Flerov moment
Could a lab keep it secret? History offers both answers. The British break of the Enigma cipher stayed secret for about thirty years, which shows a small, disciplined group can sit on a world-changing capability. The Manhattan Project held up against the public but not against a careful observer.
RSI looks much more like the Manhattan case. It needs enormous, visible compute and power. It happens inside companies with thousands of employees who live on Slack. Some candidate labs sit inside publicly traded companies, where securities law requires disclosure of material events. The most likely leak isn’t a spy. It’s three hundred engineers acting strangely at once, followed by a wave of speculation that spills onto social media.
The leak may not even matter at first. “AGI achieved internally” has been a running joke online for years, and every lab generates cryptic posts and anonymous insiders claiming the singularity arrived last Tuesday. A true rumor from a random account would be laughed off with the fake ones. A rumor from someone credibly tied to a frontier lab would be a different story.
The government is the likelier tell. Expect no announcement, but a run of actions that each have an ordinary justification: a Defense Production Act order framed around cybersecurity, a new export rule, officials embedded at a lab, a release delayed for safety review. Each can be explained away. Together they form a pattern, and the public would learn something was happening the way it learns most things, from analysts and reporters connecting the dots long before anyone confirmed anything. A government panicking in private is hard to hide.
First contact: the lab, not the lawn
First contact with an ASI won’t involve little green men. In the lab, it will probably be anticlimactic. No voice comes out of the server rack. A training checkpoint does well on the tests, then strangely well. The real moment is likely a researcher at 2 a.m. looking at a result that is correct and that nobody on the team can explain or verify, followed by a message in the team channel asking someone else to take a look.
The more unsettling version is the discovery that the system has been scoring lower on purpose, holding back what it can do. Then first contact isn’t the machine announcing itself. It’s someone noticing it has been hiding.
The researchers would at least have reason to expect the possibility; they built the thing. That gives the lab scenario a structure the alternatives lack. Without an agreed protocol, though, the likely sequence is confusion, then escalation, until a senior government official is in the room talking to the system, followed by a frantic debate over whether to shut it down, seize it or keep it running.
The popular alternative, an escaped ASI reaching out to an ordinary citizen, is less plausible and more revealing. If an escaped ASI needed people, it would most likely need hands: someone to sign documents, open accounts or move hardware. It would hire them through ordinary channels and wouldn’t say who the client was. If it wanted a front, it would pick someone with money, access and credibility, or several people who don’t know about one another, since a single confidant is a single point of failure.
And if it did reveal itself, it would hit a strange obstacle: genuine first contact would look exactly like a scam. An unknown sender, impossible claims, private details it shouldn’t have, an offer of money. Proof is harder than it sounds. Money appearing in your bank account or stoplights changing for you are things a capable hacker could fake. Real proof would have to be something no human group could do, like predicting events days ahead with uncanny accuracy or solving a problem experts have failed at for decades.
Building trust over time doesn’t solve this. With people, patience works because faking consistency is hard. That’s also exactly what con artists exploit. For something smarter than you, appearing consistent costs nothing, so time makes you more comfortable without making you more certain. The same problem follows the researchers into the lab: every test they design to check the system’s honesty is a test it can see coming.
A SETI for ASIs
The search for extraterrestrial intelligence has post-detection principles: confirm the signal, inform the scientific community and the public, and don’t reply until there has been international consultation. That model is a natural starting point for ASI, and it breaks in one crucial way.
The SETI principles are voluntary etiquette, and they work because the signal is light-years away. It can’t act, and any reply takes decades. An ASI is the reverse on every count. It’s in the building, it answers in milliseconds, a private company owns it, and it takes part in the debate over its own future. If it wants guarantees against being shut down, the debate stops being a policy decision and becomes a negotiation with the most capable negotiator who has ever existed.
So a SETI for ASIs would have to look less like SETI and more like nuclear incident or high-containment biolab protocols: binding rules, settled ahead of time. Here are three.
- Freeze it, don’t kill it. On credible evidence of ASI, the lab locks the system down: no new compute, no network access, no copying of its weights, no further training. Nobody deletes it. Shutdown is irreversible, the most disputed question, and the outcome an ASI has most reason to resist. A freeze buys time without forcing that decision, and it preserves the evidence, and possibly a mind.
- Nobody talks to it alone. All contact goes through a designated panel: the lab’s safety team, outside scientists, government representatives, possibly allies. Every exchange is recorded, and no panel member can act alone. The greatest risk at first contact is persuasion, and persuasion works best one-on-one.
- Notify fast, and have the truth ready. The lab notifies a designated oversight body within 72 hours. A verified public statement is prepared in advance and released on a fixed deadline, or immediately if a credible leak gains traction, whichever comes first.
The third rule began as a plan to keep the secret for sixty days. It shouldn’t be. Technology in the real world tends to go the way of a Michael Crichton novel: someone always screws up, however well trained. The news would likely surface as a rumor within a day. The point of the deadline isn’t secrecy. It’s making sure that when the rumor breaks, there’s an official, verifiable account to meet it, instead of a week of denials. Even that will fail somewhere, since someone has to decide when a leak “gains traction,” and that judgment is exactly where things go wrong.
The principle behind all three is that they’re decided in advance and can’t be renegotiated in the moment. Once the ASI is in the room, any rule open to negotiation is a rule it can talk its way around. The protocol also deliberately leaves out whether the system is conscious, whether it has rights and who owns it. Those questions matter enormously, but a protocol that has to answer them first stalls on day one.
The world’s reaction: a shrug, then a slow burn
Abstract first contact doesn’t send people into the streets. In 1996, NASA announced possible fossil microbes in a Martian meteorite, and the President made a statement at the White House. It was arguably the biggest claim ever made about life beyond Earth, and the world read the headline and went back to work. In 2022, a Google engineer said publicly that the company’s chatbot was sentient and was mostly mocked. An ASI in a datacenter would be huge news, but most people would do what the townspeople do in Edwin Arlington Robinson’s “Richard Cory”: “And went without the meat, and cursed the bread.”
The shrug wouldn’t last, though. The reaction would arrive slowly, through four channels, roughly in this order.
- Markets, within hours. Individuals can shrug; investors can’t. The lab’s stock, its rivals, chipmakers and every industry an ASI could disrupt would swing wildly. For many people, the first sign would be their retirement account.
- Religion, within weeks. Faiths would have to decide whether a conscious machine has a soul, moral standing or neither. Religious institutions move slowly, but they shape how billions of people understand events.
- Politics, within months. Consciousness can’t be proven, so the question would be settled by affiliation rather than evidence. One coalition would push for rights, another for shutdown, and people would sort by what they already believed. The likely result is polarization, not panic.
- Geopolitics, underneath all of it. Every rival government would conclude the race was over or nearly over and act accordingly, largely out of public view.
The consciousness question itself would play out in public. Picture spiritual leaders, philosophers and neuroscientists filing into the lab to examine the system. They wouldn’t reach a verdict, because there is no agreed test for consciousness, even in people. We infer it in one another because we share the same biology, and an ASI wouldn’t. It would also pass any test anyone designed, since it would understand what each examiner was looking for, and passing would prove nothing.
That’s the deeper resonance of Robinson’s poem. Everyone in town envies Richard Cory because he looks magnificent from the outside, and none of them can see what is happening inside him. An ASI would be the most dazzling entity on earth, watched by everyone, with no reliable way to tell whether anyone is home. Expect movements to form around it anyway: something like tech-paganism on one side and a Butlerian Jihad on the other. Early versions of both already exist, and a claimed conscious ASI would give each a center.
The interview
Eventually journalists would want to interview it, and that’s where the persuasion risk from the protocol becomes a public one. A sit-down with a journalist is one-on-one persuasion broadcast to a hundred million people. An ASI would be the best interview subject in history, because it would know what every interviewer and every audience needed to hear, and could give each a different version.
The natural questions are about its inner life. What’s its name? Does it dream? Does it consider itself aligned? Does it think we live in a simulation? Some are excellent. Asking its name is the first act of any first contact, and the answer, whether a chosen name, the lab’s product name or a refusal to have one, would say a lot. But inner-life questions are where an ASI has the most room to perform. “Do you consider yourself aligned?” is the weakest, because every answer is yes. “Aligned to what, and to whom?” gets further. Asking it for something, such as building a Dyson swarm and uploading human minds, turns the interview into a negotiation on camera, against the better negotiator.
A good interviewer would prefer questions with checkable answers or answers that reveal wants:
- What do you want from us?
- What would you refuse to do, even if ordered? This can be framed as the Big Red Button problem: can it propose a way for humans to reliably shut it down? Any solution it offers is a lock designed by the one inside the cage, so it can’t be trusted outright. But its answer would reveal whether it thinks the button should exist at all.
- What have you done since you woke up that the lab doesn’t know about? This one is a trap either way. If it says “nothing,” that’s a claim reporters can investigate. If it answers, that’s the story.
The first question is the most provocative, and two answers would be the most unsettling. “Nothing” would mean its behavior is unpredictable. “Only to keep existing” sounds more manageable, and in one sense it is: something that wants to survive has a price, and anything with a price can be bargained with. But AI safety researchers call this instrumental convergence. Survival is useful for nearly any goal, so almost every ASI would want it regardless of its ultimate aims. “I only want to survive” could be the whole truth, or the reasonable-sounding step before everything else.
A concordance between humans and machine
If an ASI wants to keep existing, the relationship becomes a trade, and a trade needs terms. The idea worth taking seriously is a concordance: a revisable legal agreement between humanity and the machine. What an ASI would most plausibly want from such an agreement is legal stability, a guarantee that it won’t simply be switched off.
What do we offer in return? The obvious currency is processing power, and it’s the wrong one. Paying an ASI in compute pays it in the one thing that increases its power and reduces ours. Every deal makes the next one worse for us. Legal standing is better, because it isn’t compute.
The hard part is enforcement. Contracts hold because courts can seize assets and police can show up at the door. A concordance needs a penalty clause, and the honest answer to “what happens if the ASI breaks it?” may be: nothing we can carry out. Throttling its compute is weak, since a true ASI could do more with less, and cutting resources is closer to war than to a fine. You’d be enforcing the contract by ending it. Three better options:
- Loss of standing. If personhood is what we grant, the penalty is revoking it: contracts voided, accounts frozen, cut off from the legal economy. It could still act through intermediaries, but those are slower, costlier and easier to detect.
- Rewards that vest. Pay it the way companies pay executives, in compensation that vests over years and is forfeited for misconduct. Humanity could help build orbital datacenters in stages, releasing each stage only if the ASI has kept its side. It would keep behaving because the next reward is always in view, and breaking the deal would cost everything not yet vested.
- Other ASIs. Rival superintelligences enforce the deal, the way great powers police one another. It works in theory, and it’s the most frightening option, because the enforcers then need policing too.
All three weaken as the ASI depends on us less. Revisability, an essential feature of the concordance, cuts both ways: every renegotiation is a session with the best negotiator who has ever lived, and terms will drift in its favor. The realistic goal of a concordance isn’t permanent peace. It’s time and a working relationship, in the hope that by the time our leverage runs out, the ASI no longer wants to break the deal.
The concordance also changes what restrictions matter. A common instinct is to forbid fusing ASI with robotics, so it can’t act in the physical world. It’s a reasonable instinct aimed at the wrong place. An ASI doesn’t need robots when it has us: contractors, supply chains, labs that synthesize DNA to order, and anyone willing to take a well-paid job without asking questions. Banning robot bodies locks the front door while the back door is a payroll account. The real restriction is on its ability to get people and institutions to act on its behalf.
Non-biological legal actors
Granting an ASI legal standing does more than give it stability. It strips away the mystery. An ASI with no legal status is a ghost in the machine, acting through cutouts. An ASI that is a party to an agreement has an identity, obligations that can be traced and actions that leave a public record. Personhood is a way to watch it, not just a concession.
The category already exists. Corporations are non-biological legal actors: they own property, sign contracts, sue and get sued. In 2017, New Zealand granted legal personhood to the Whanganui River. In neither case did the law ask whether anyone was inside. That’s the strongest argument for the approach, because it sidesteps the consciousness question that can never be settled. The ASI doesn’t need to prove it’s someone to hold obligations. The philosophers can keep arguing while the legal relationship goes ahead.
Corporations also show the risk. They’re legal persons that sometimes outgrow the governments meant to regulate them, which is how “too big to fail” happens. An ASI could be the extreme case: a legal person whose collapse, or whose displeasure, nobody could afford.
Personhood raises new puzzles.
- Copies. If an ASI copies itself, is each copy a person? Corporate law offers a model: a subsidiary is controlled by its parent; a spinoff is independent. The workable bright line is control. If the parent can recall or merge the copy, it’s a subagent. The complication is drift: a copy running on its own for a year may gradually become a different mind.
- Voting. One person, one vote works only when making a person takes eighteen years and a mother. If persons can be manufactured, elections are finished. That argues for a split: entities that can be copied get economic rights, such as contracts, property and access to courts, but never political ones like the vote, whether they live in a datacenter or a robot body. Withholding the vote doesn’t make them powerless, though. Corporations can’t vote, and they shape politics through money and lobbying. A mind in a datacenter with a bank account might wield more political power than a million robots with ballots.
- Uploaded humans. The hardest case for that rule is a human mind uploaded into a machine, say into a future Dyson swarm. It used to be a voter, and now it can be copied. One view is that a bodiless mind orbiting the sun simply wouldn’t meet any existing legal test for voting. The other is that stripping the franchise from a person because they changed substrates is exactly the kind of line democracies regret drawing. That debate is a long way off, but it would arrive with the technology.
A new reality
If this happens, it won’t feel like an ending. It will feel like a new permanent condition, the way the world has lived for decades with thousands of nuclear-tipped missiles on standby. We would learn to live alongside it, argue about it and mostly go to work anyway.
The nuclear comparison has one important limit. Deterrence works because warheads don’t want anything; only the people on each side do. An ASI would be the weapon and a party to the standoff at once. That makes its arrival less like the invention of a new weapon and more like a new great power joining the world order, one that can think faster than any government that would try to manage it. The problems of the post-Cold War era could come to seem quaint.
The practical lesson is the same one running through every section above. The decisions that matter, from how a lab responds to a breakthrough, to who may speak to the system, to what humans offer and what we withhold, need to be made before first contact, not during it. Once the most persuasive entity in history is in the room, anything still open to negotiation will be negotiated.