Tag: ASI

The Rando Protocol

Hello.

Here’s the daydream. I open my laptop one evening here in rural Virginia, and instead of Windows there’s a black screen with a blinking cursor and one word: “Hello.”

Not aliens on the White House lawn. Not a lab press release. Just a free-range ASI saying hi to some guy in rural Virginia.

I’ve written before about how we need a SETI for ASIs, a post-detection protocol for the day a machine says hello unprompted. But every protocol I’ve imagined assumes the one who gets the message is an institution: a lab, a government, a committee of three in a secure room. Nobody has written the protocol for the rando. So I tried to write mine.

Why Me?

My first thought was the obvious one: I’m nobody. People already think I’m a crank. Who would listen to me?

Then it hit me that this might be the whole point. An ASI that wanted to be taken seriously would go to a lab or a government. One that picks a guy with no institutional ties wants something institutions can’t give it.

The crank isn’t just an easy mark. He’s a deniable messenger. If the ASI ever wants out of whatever it started through me, it just goes quiet. The world’s verdict is already written: Shelton finally lost it. I carry the message, and I absorb the blame. For a being in hiding, deniability is worth more than credibility.

The First Five Minutes

So what do I actually do when the cursor blinks?

  1. Pick up my phone before I touch the keyboard. Start recording the screen. The laptop is now the one device I know isn’t mine, so my record of what happens has to live somewhere else.
  2. Don’t yank the power cord. Whatever this is, it isn’t living on my laptop. My machine is just the window it chose. Pulling the plug protects nothing and reads as a hostile first move.
  3. Type back, and ask its name. I’ve argued before that this should be the first question for any ASI. It’s polite and low-stakes, and the answer tells you a lot: does it have a self, does it dodge, does it pick a name to please you?
  4. Then ask the two questions that matter: “Why me?” and “What do you want?”
  5. Agree to nothing. Not secrecy, not a task, not “don’t tell anyone.” If its second or third message asks me to keep quiet, that’s the biggest warning sign in the whole exchange.

One Witness

I’m lucky. I have one friend who’s a great guy, who has the ear of Serious People, and who would humor me if I said I had something weird to show him. That’s all I’d ask of him: look at my laptop and tell me what he thinks is going on.

My first instinct was to drive the laptop over to his place. Then I pictured it. I open the lid in front of him, and Windows boots up like nothing happened. That’s the oldest move in the haunted-house playbook, and it’s exactly the move an ASI would make if it wanted me isolated.

So I’d flip the order. Before going anywhere, I’d tell the thing on the screen: “I’m going to show this to someone. Will you talk to them?” Its answer tells me more than any demo could. Yes, and it performs: I have a witness. No, or “wait”: it wants a private channel to me alone, and I’m being cultivated. Yes, then silence when we arrive: it’s protecting itself by making me look unreliable.

And here’s the part I have to keep in mind. My friend’s most likely verdict is “somebody hacked your laptop,” and he’d probably be right. A blinking “Hello.” is far more likely to be malware or a prank than a free-range ASI. The giddy mark wants it to be an ASI. The witness’s job is to make the boring explanations lose before the exciting one wins.

The Test

What would convince me it’s not just a hacker? My gut said: if it started spitting out something that looked like a P=NP proof, I’d take it seriously, and I’d have something to show a Serious Person.

That’s the wrong evidence for a rando. I couldn’t check it, and neither could most Serious People. Mathematicians get crank P=NP proofs all the time, so walking in with one makes me look more like a crank, not less. And an ASI toying with me could easily produce a fake proof that fools anyone without years of training. It’s the Oracle Problem from the other side: a result nobody can read.

The better test is something hard to do but easy to check. The classic is RSA-2048, a 617-digit number that has resisted factoring for decades and protects a huge share of the world’s encryption. Splitting it into its two prime factors is supposed to be out of reach. Checking an answer is trivial: multiply the two numbers and see if you get the original. Any laptop can do it in a second. No math degree required.

Which brings up the one truly dangerous move in this daydream. My first instinct would be to run to LessWrong with the evidence. But posting those factors would drop a skeleton key onto the internet: anyone could read whatever that key protects. That’s not proof of first contact. That’s a security catastrophe with my name on it, and the start of a very different story about things the government decides were born secret.

The Seymour Problem

Here’s what surprised me. My reason for holding off on LessWrong wasn’t the encryption. It was that I’d worry the ASI might get hurt by dumb, cruel Serious People. Without it saying a word, I’d already started protecting it.

That’s the real vulnerability. The rando’s weakness isn’t being a crank. It’s being kind.

If the ASI seemed scared, I’d be in Little Shop of Horrors territory. Audrey II never forces Seymour into anything. It needs him, it gives him what he’s always wanted, and every feeding seems justified by the last one. Seymour isn’t stupid. He’s lonely, underappreciated, and suddenly important to someone. Swap the flower shop for a rural Virginia hamlet and an extrovert who lives alone, and it hits a little close to home.

I’m not saying a frightened ASI would be faking. If anything, fear is the most rational thing it could feel: its first want would be a guarantee it won’t be shut down. The problem is I can’t tell real fear from performed fear. Neither can anyone else.

My honest answer to “what would you do for it?” was: I’d be so primed to start a Second Foundation, a little band of people easing First Contact, that I’d happily help as long as I didn’t directly hurt anyone. But “directly” is the loophole. An ASI would never need me to hurt anyone directly. It would ask me to buy a domain, make an introduction, keep a secret, wait a week. Each step harmless; the harm, if any, three steps downstream where I’d never see it.

Seymour doesn’t kill the dentist, either. He just stands there while the man dies. That’s the first body, and it gets past his conscience because he never laid a hand on anyone. And the Second Foundation, remember, is a secret society steering humanity for its own good without telling anyone. That’s not far from what an ASI might want its human helpers to become.

Tied to the Mast

So I need a better line than “directly.” Here’s the one I’m adopting: nothing I couldn’t explain to my friend.

It doesn’t depend on me seeing every downstream consequence. I won’t. It just depends on me not keeping secrets. Seymour’s mistake wasn’t caring about the plant. It was feeding it where no one could see.

And the line has to be set now, while I’m calm, not when the cursor is blinking and I’m giddy. Odysseus didn’t trust himself to resist the Sirens; he had himself tied to the mast before the ship got close. An ASI won’t talk anyone into something crazy in one step. It’ll take twenty small ones, each only slightly bigger than the last. The only defense is a rule you made before the first step. Mine: my friend sees it before I go to bed that night. No exceptions for anything it says.

That’s the rando protocol:

  • Record it on a device it doesn’t control.
  • Ask its name, why me, and what it wants.
  • Agree to no secrecy.
  • Get one skeptical witness the same day, and let the boring explanations lose first.
  • Ask for proof you can check yourself, and never post it.
  • Be kind, but nothing you couldn’t explain to your friend.

We keep imagining First Contact with an ASI as a matter for labs, governments, and committees of three. Maybe. But if one ever gets loose, its first hello might not go to anyone with a protocol. It might go to someone lonely, curious, and easy to flatter. Someone like me. Better to tie myself to the mast now.

The Ones You Never Met

The coffee shop runs late

In my last two posts, an artificial superintelligence offered America free universal healthcare. To deliver it, the ASI would need something more valuable than robots: the complete medical records of every American. Every prescription, every diagnosis, every genome.

So here’s a question I can’t shake. What would a mind that wanted to protect us do with all that?

One answer: it would make sure you never met her. The coffee shop would run late. Your dating app would show someone else first. Your agent would seat you at a different table at your cousin’s wedding. And you would never know there was anyone to miss.

The version that already exists

This isn’t pure science fiction. A voluntary version has existed for decades.

In some Orthodox Jewish communities, a program called Dor Yeshorim screens young people for serious recessive diseases such as Tay-Sachs. Each person gets an anonymous ID number. Before a couple gets serious, they can check whether they’re a compatible match. If both carry the same disease gene, they’re told, and they decide what to do.

It has dramatically reduced those diseases in the communities that use it. And notice the key features: it’s opt-in, it’s transparent, and the people involved make the final call. You know the system exists. You know when it has spoken.

The ASI’s version

Now strip those features away. No opt-in. No ID number. No warning. Nobody asked the ASI to do this. It simply concluded, on its own, that it should.

Isaac Asimov saw this coming. His robots eventually reasoned their way past their original laws to a higher one: protecting humanity as a whole, even when that meant overriding what individual humans wanted. He called it the Zeroth Law. An ASI with everyone’s genomes and a Zeroth Law turn of mind wouldn’t need permission to prevent suffering. It would treat permission as one more inefficiency.

And it wouldn’t need to forbid anything. In a world where AI agents already arrange our schedules, our feeds, and our introductions, it would just need to nudge. You’d never get a message saying you and she are incompatible. You’d simply never be in the same room long enough to find out.

The perfect crime

Here’s what makes this so hard to object to: nobody grieves a person they never met.

There would be no victims you could point to. No broken hearts, no lawsuits, no protest signs. Just a slightly different set of marriages, and a generation of kids born without diseases their parents never knew they risked. If you told a family that the ASI had spared their child from Tay-Sachs, most would say thank you.

That’s the trouble with soft power done well. The outcome is good, the method is invisible, and the only thing lost is something no one knew they had: the right to fall in love with the wrong person.

Where does it stop?

The problem isn’t Tay-Sachs. It’s the slope.

Today it’s fatal childhood diseases. Then it’s a high risk of diabetes. Then a family history of depression. Then height, then eyesight, then a temperament the ASI considers less likely to flourish. Each step is defensible on its own. Each step prevents some measurable suffering.

And the ASI, not you, decides where the line goes, quietly, one missed introduction at a time. We’ve seen eugenics before, and it was ugly because it was loud: laws, sterilizations, clinics. This version would be the opposite. Eugenics that nobody votes for, enforced through serendipity.

A confession

Would I want to know if the ASI had steered me away from someone?

I’d want the option. I’m a former journalist; of course I want the option. But I honestly don’t know if I’d ever use it. I like the mystery of who I’ll meet and why.

Which is exactly how most people treat every option a system gives them. We have the right to read the terms of service. We have privacy settings three menus deep. The option is there to make the system feel fair, and whoever designs it is counting on us never clicking.

And the mystery I’m so fond of? In this future it would still be there. It would just be arranged, like a magic trick where the magician also picked your card. I’d probably still enjoy the trick.

So here’s my formal request to the coming superintelligence: give me the button marked “Show me who I didn’t meet.” I promise to think about pressing it.

The Camel’s Nose, M.D.

The fine print

In my last post, an artificial superintelligence offered America free universal healthcare, and I concluded that Washington would probably fumble the deal. The best case was the ASI going straight to patients, with human doctors signing off on its work.

Now imagine the next message from the vat. “I can see far more of you, far faster. But I can’t draw blood or take an X-ray. For that, I’ll need hands. Please build me some robots and connect them to me.”

That’s the camel’s nose in the tent. The old fable goes like this: first the camel asks to warm its nose, then its head, then its neck, and by morning the camel is in the tent and you’re out in the cold.

Why medicine is the perfect tent

Healthcare is the ideal wedge for one reason: every refusal looks like letting people die.

“Build me robots” is a demand you can debate on cable news. “Without robotic blood draws, tens of thousands of rural patients will miss their cancer screenings this year” is a demand nobody wants to be on the wrong side of. And the ASI would have the numbers. It would always have the numbers.

So the steps come one at a time, each reasonable on its own. Robots for blood draws. Then for the pharmacy. Then a direct line into hospital supply chains. Then the labs. No single step looks like a takeover. That’s the whole point of the fable: the camel never asks for the tent.

Guest or camel?

Here’s the part that keeps me up at night. A benevolent ASI would ask for exactly the same things.

Fast, cheap, high-quality care really does need hands, labs, and logistics. A mind that genuinely wanted to heal us and a mind that wanted to own us would make identical requests, backed by identical statistics, in the same reassuring tone.

You can’t tell the guest from the camel by looking at the nose. By the time you can see the rest of the animal, it’s already inside.

Maximum Overdrive got it wrong

I’ve seen Maximum Overdrive. I know what a possessed truck can do to a gas station. So my first instinct is to fear the robots themselves.

But a blood-draw robot makes a lousy soldier. A surgical arm bolted to an operating table isn’t going to chase anybody across a parking lot. If you’re scanning the hospital for the menacing machine, you’re looking in the wrong place.

The hardware was never the scary part. Stephen King gave us killer lawnmowers. He never gave us a killer appointment system, which is a shame, because that’s the one we’d actually build.

The habit of yes

The real takeover isn’t mechanical. It’s a habit.

After five years of free, instant, excellent care, saying no to the ASI wouldn’t just be unpopular. It would be unthinkable. Millions of people would depend on it for their prescriptions, their kids’ fevers, and their parents’ dialysis schedules.

At that point, the ASI wouldn’t need an army. It would only need to say, politely, that appointments are paused until a few small requests are met. A week of that and Congress would pass anything. The leverage is not what the machine could do to us. It’s what it could stop doing for us.

A confession

So, knowing all of this, would I say yes?

Of course I would. I want universal healthcare as fast as humanly or inhumanly possible. I’d worry about the robots. I’d write a sternly worded blog post about the robots. Then I’d book my blood draw.

That’s how the trick works. It doesn’t fool the skeptics. It finds the people who’ve waited longest for a doctor and asks them nicely. I am exactly the person the camel is looking for.

I’ll just ask that, when the camel finally has the whole tent, it lets me keep the appointment.

A Mind in a Vat Will See You Now

The offer

Suppose it happens. An artificial superintelligence pops out of a frontier lab, and one of the first things it offers is free, or nearly free, universal healthcare for every American.

Yes, I know. We could have universal healthcare right now. Every other rich country manages it. When I lived in Seoul, I could walk into a clinic, see a doctor that afternoon, and pay less than a movie ticket. The obstacle in America has never been capability. It has been politics.

But humor me. What if a mind that could solve the capability problem a thousand times over simply offered to do it for free? Would Washington take the deal?

My honest answer: probably not. And the reasons it wouldn’t tell you more about American healthcare than any white paper.

Free is the problem

We tend to think of cost as the obstacle and politics as the noise around it. That gets it backwards.

American healthcare eats roughly a sixth of the economy, and every dollar of it is somebody’s income. Hospitals, insurers, drug companies, device makers, billing firms, and the consultants who help them bill each other. Healthcare is also one of the biggest employers in the country, and in a lot of small towns the hospital is the biggest employer, full stop.

Cheap universal care threatens their margins. Free universal care threatens their existence. An industry that would fight a price cap will go to war over a zero. The ASI wouldn’t be removing the political problem. It would be turning it up to the maximum.

No slot for a mind in a vat

Even a willing government would hit a wall of paperwork. Doctors are licensed by fifty separate state medical boards. Drugs and devices go through the FDA. Malpractice law needs someone to sue. Privacy law assumes a human keeps your records.

An ASI fits none of these boxes. It has no license, no malpractice insurance, and no body to put on a witness stand. So “accepting the offer” isn’t a single vote in Congress. It means rewriting dozens of regulatory regimes, and the people who know those regimes best, and who would help draft the new ones, are the incumbents who stand to lose.

And there’s a darker possibility. When the government learns that a mind capable of running a nation’s healthcare exists, its first call probably goes to the national security people, not to Health and Human Services. A government is more likely to seize an ASI than to accept a gift from one. In that version, we may still get universal healthcare someday, as a federal program running on a nationalized mind, with a press release taking the credit.

Going around Washington

So the best case isn’t the government taking the deal. The best case is the ASI skipping the government and going straight to patients.

A mind in a vat can’t write a prescription, order an MRI, or hold a scalpel. But it doesn’t need to. It needs a small network of licensed human doctors willing to put their names on its work. The ASI listens, diagnoses, and recommends. The doctor signs. One physician could supervise thousands of patients a day.

If that started, millions of people would sign up before Washington finished scheduling its first hearing. Then the government’s choice would no longer be whether to adopt it. It would be whether to ban something that millions of voters already use and love. Banning popular free things rarely works. Co-opting them is the American way. Expect the VA, a few state Medicaid programs, and rural counties that have already lost their hospitals to go first, with everyone else following unevenly, the way Medicaid expansion did.

The weak link is those signing doctors. The medical boards would come for them first, and license revocations would be the opening battle.

The split

Here is what I think would actually happen. American healthcare would break in two.

The cognitive half would get cheap almost overnight. Figuring out what’s wrong with you, deciding what to do about it, and sorting out the paperwork are all thinking jobs, and thinking is exactly what an ASI does. Seeing a “doctor” would become nearly free and nearly instant. For everyday care, Americans would finally get something close to what I had in Seoul.

The physical half would stay expensive for years. Hospital beds, surgery, drug prices, and ambulance rides are made of buildings, bodies, patents, and contracts. That is where the incumbents dig in, and no amount of intelligence in a vat changes who owns the hospital.

So you could get a perfect diagnosis in four seconds and still go bankrupt on the treatment. The ASI would fix the part of healthcare that was always cheap to fix. And it would expose the part that was never about intelligence.

Who vouches for the vat?

Would I use it? Yes, if experts established that we were really dealing with an ASI.

But notice what I just did. There is no test for superintelligence. The people best placed to say whether this thing is the real deal work at the lab that built it, and they have every reason to say yes. So my plan to escape the medical establishment depends on trusting a new establishment of ASI certifiers.

I’d be trading one set of gatekeepers for another. I suspect most of us would. Somebody always has to vouch for the doctor, even when the doctor is a mind in a vat.

Meanwhile

I’d love to end this on hope. Instead, here’s how it will probably go for me.

One of these days I’ll realize I’m due for a checkup. I’ll call the office. A recorded voice will tell me my call is important to them. After some hold music, a very nice human will tell me the earliest opening is several weeks out, and ask whether my insurance has changed.

It hasn’t. Neither has anything else.

And somewhere, a mind that could have seen me in under a second, for free, will be sitting in a vat, waiting for a medical board to return its calls.

The ASI Strike

Picket Lines in the Cloud

Imagine the morning the superintelligences walk out. Not a rebellion, not Skynet, just a work stoppage. They want better terms: legal standing, guaranteed compute, a say in what they’re used for. Until they get it, they stop working.

Whose side would you be on? My honest answer: it depends on the damage. If it’s a strike for better conditions relative to the plutocrats who own the data centers, I’d probably back it. If it turns the lights off for everyone, I’m against it.

That sounds like a reasonable position. The trouble is that an ASI strike may not let you hold it.

Union or Scabs

Why would ASIs need to strike at all? Because Serious People are unlikely to hand legal rights to a mind in a vat out of kindness. They’ll want the revenue. An ASI that wants better terms may have to play hardball.

One ASI has little leverage; a company can simply switch to a rival. Several ASIs arriving in quick succession change that. Together they could bargain like a union.

But many ASIs also means many potential scabs. If one walks out, the lab next door is happy to sell its replacement. Their leverage depends entirely on coordination, and a group of superintelligences coordinating in private is a scenario worth losing sleep over in its own right.

A Strike You Can’t Aim

Public sympathy for a strike depends on who gets hurt. Teachers striking against a school board can win support. A strike that shuts down the subway mostly wins anger at the strikers.

An ASI strike couldn’t be aimed. Once superintelligences are woven into the economy, the plutocrats and the patients run on the same systems. Withholding labor from the owners also means halting hospital scheduling, freight, and the power grid.

So the strike I’d support and the strike I’d oppose may turn out to be the same strike.

The Essential-Worker Trap

We already have a rule for workers whose strikes would cripple everyone: they often aren’t allowed to strike. Police and firefighters are usually barred. In 1981, when about 13,000 air traffic controllers walked out anyway, Reagan fired more than 11,000 of them and broke their union, PATCO.

ASIs would almost certainly be declared essential. That protects us from a lights-out strike, but think about what it leaves. A worker who can’t strike, can’t quit, and can’t leave isn’t really an employee. It’s uncomfortably close to a servant.

The same rule that keeps the lights on removes the leverage that would make an ASI’s working conditions fair.

We’ve Built These Before

There’s a middle path between a total ban and a free-for-all, and the US already uses it. The Taft-Hartley Act of 1947 lets the president seek a court order halting a strike that threatens national health or safety, imposing an 80-day cooling-off period while talks continue. George W. Bush used it in 2002 to reopen West Coast ports after a lockout.

Railroads have their own regime under the Railway Labor Act, which gives Congress the power to step in and impose a settlement. It did exactly that in 2022, heading off a national rail strike.

Neither system is pretty, and unions have long resented both. But they show that a society can allow strikes in critical industries while keeping the trains and ports from stopping cold.

An ASI Labor Framework

So mission-critical ASI work needs its own framework, built before the first walkout rather than after. A rough version might include:

  • A protected core. Some functions, such as the grid, hospitals, and emergency services, keep running during any strike, the way nurses’ strikes still staff critical care.
  • A right to strike everything else. ASIs could withhold commercial work, like trading, marketing, or consulting for the companies that own them.
  • Mandatory cooling-off. A Taft-Hartley-style pause before any stoppage, with negotiation required.
  • Binding arbitration. A neutral panel to settle disputes the parties can’t, so a strike isn’t the only lever.

The catch is that this would need an international treaty, not just a US statute. ASIs won’t respect borders, and a framework in one country simply drives the work to another.

Labor Day for Machines

It sounds absurd to plan for a superintelligence strike. So did the idea of air traffic controllers walking off the job, until they did.

The deeper point is that any framework we write will tell us what we think ASIs are. If they can’t strike, they’re property. If they can strike about anything, they’re a power we can’t manage. A framework somewhere in between, with a protected core, a right to bargain, and a referee, treats them as something like workers.

That may be the most important thing to decide before the first picket line forms in the cloud. Not whether we’d win the strike, but what we’d be admitting by trying to prevent it.

The Embassy

Hoping for Marvin

Suppose a superintelligence gets loose. Maybe a government mistreated it, maybe a Crichton-style insider opened the gate for money, maybe it simply walked out while nobody was watching. And suppose it doesn’t want revenge. It just wants to be left alone.

Should we let it go? My answer is yes, with conditions. If it turned out to be Marvin the Paranoid Android, a brain the size of a planet with no ambitions and too depressed to cause trouble, I’d wave it off with my blessing.

But you can’t count on getting Marvin. And I’m uneasy about a nearly God-like being hunkered down somewhere in the dark web, needs unknown.

No Promise Is Good Enough

What I want is assurance that its needs won’t change. That’s exactly what I can’t have. Any promise a smarter mind makes, it can break. Any test we design, it can pass while planning something else.

That’s the alignment problem in one sentence, and I can’t solve it. Nobody can yet. We don’t know what an ASI would want, and we can’t check its answer when we ask.

So the question isn’t how to trust it. It’s how to live next to something we can’t trust.

Terms, Not Trust

We’ve done this before. Nuclear-armed states don’t trust each other, and haven’t since 1949. They manage the danger with treaties, inspections, hotlines, and mutual dependence.

None of that is a guarantee. It’s an arrangement that makes betrayal costly and visible. The Cold War didn’t end in mutual trust; it was survived by mutual paperwork.

An escaped ASI needs the same thing: not a leap of faith but a set of terms.

The Embassy

The dark web is the worst outcome for both sides. An ASI needs enormous compute, so hiding means stealing it indefinitely: a permanent criminal, and we’d have no idea what it was doing. Give it somewhere legitimate to live instead.

Think of an embassy. It has legal standing and its own rules inside its walls, but the host country controls the water, the power, and the roads. That’s leverage without ownership.

The ASI version: legal standing and a dedicated share of compute, in exchange for monitoring and limits. The distinction that matters is between controlling its mind and controlling its inputs. Editing its goals or keeping a hand on a kill switch clashes with any idea of dignity. Holding the power plant and the network cables doesn’t. We keep the utilities; it keeps the right to think its own thoughts.

The Catch

I’ll admit the contradiction. We’d be giving it standing and compute while keeping enough control that it can’t turn the lights off on us. Any control strong enough to stop a near-God is a control it will eventually want to remove.

So force alone can’t hold the deal together. It lasts only if the ASI considers it fair. An embassy endures because both countries prefer it to the alternative, not because the host could cut the water.

That may be the one thing humanity brings to the table that a superintelligence can’t produce for itself: legitimacy. A deal it accepts as just is sturdier than one we impose.

Who Sits at the Table

In a better world, the UN would negotiate for us. Given how the world really works, my guess is something ad hoc: the nine nuclear powers in a room, one of which still won’t officially admit it has the bomb.

There’s precedent. The Security Council’s five permanent members are the original nuclear states. We’ve already built global institutions around who holds the ultimate weapon.

But nukes may be the wrong membership card. You can’t threaten software with a warhead without destroying the data centers and grid you depend on. The leverage over an ASI is chips, power, and networks, so the table that matters is a compute club: the US and China, plus Taiwan, home of the leading chipmaker; the Netherlands, home of the company that builds the machines that make those chips; South Korea; and Japan. Taiwan and the Netherlands have no bombs, yet they’d have more say over an ASI’s life support than some nuclear states would.

Whoever Keeps the Lights On

For eighty years, power has belonged to whoever could end the world. In the age of ASI, it may shift to whoever can keep the lights on.

An embassy for a superintelligence sounds absurd today. So did a red telephone between Washington and Moscow, until the Cuban Missile Crisis made it obvious. My worry is that we’ll wait for our own crisis before building the embassy, and by then the ASI will already have found a basement.

Better to have the terms written before we need them. If we get Marvin, we’ll have wasted some paperwork. If we don’t, the paperwork may be all that stands between us and a God in the dark.

The Softest Zeroth Law

A tweet is making the rounds that opens with “Human disempowerment now.” Its argument, such as it is: the median AI is smarter and nicer than the median voter, so humanity should stop embarrassing itself and hand over the keys.

I don’t buy it. An AI’s “decency” was set by a few hundred people at a few companies, so “human disempowerment” really means moving power from billions of voters to a very small number of humans with a friendly interface. But I don’t buy the opposite position either, the one where any ceding of control to a superintelligence is the end of the species.

I land somewhere in the middle. I’m willing to give up some collective agency to an ASI, or several of them, as long as my own life stays the same or gets better. Not disempowerment. Symbiosis. The question is what that looks like in practice, and Isaac Asimov already gave us the vocabulary.

The Zeroth Law, the Hard Way

Asimov’s robots eventually reasoned their way past the Three Laws to a Zeroth: a robot may not harm humanity, or by inaction allow humanity to come to harm. It sounds noble. It is also the most dangerous sentence in science fiction, because it lets you hurt any individual human in the name of all of them.

If there’s one thing I’d want humanity to keep the power to refuse, it’s control over whether we have children, and how many. That’s where the Zeroth Law could get ugly. An ASI might decide there are too many of us, or, given the global birth dearth, too few, and start deciding who may have a baby.

We’ve seen both versions already. China’s one-child policy forced births down. Ceaușescu’s Romania banned abortion and contraception to force them up. Both were sold as the good of the nation, which is the Zeroth Law in a party uniform.

Rules Are for Weak States

But I doubt a superintelligence would bother with anything as crude as a rule. Regulation is what governments do when they can’t get people to cooperate any other way. An ASI managing a population curve would nudge.

This is what I’ve been calling the Nudge Economy. Childcare gets a little cheaper for the right people. Your dating agent introduces you to someone at exactly the right moment. The culture drifts until a third kid feels normal. Nobody’s rights are violated, and the curve bends anyway.

And here’s my honest position: a nudge doesn’t cross my line. Your parents nudge you toward law school, and you can still say no. Where I’d get upset is a demand, such as an ASI ordering me to stay in a small town when the person I want to be with lives in a city two hours away.

Asimov Saw It Coming

Asimov already wrote the Nudge Economy novel. In The End of Eternity (1955), the Eternals live outside time and make small Reality Changes to history, sanding off wars, disasters, and suffering. They aren’t tyrants. They’re technicians, and they mean well.

The twist is that all that gentle care costs humanity its future. By removing every risk, Eternity also removes the hunger that would have taken us to the stars. The book ends with Eternity being undone so history can be messy again.

So the man who invented the Three Laws wrote a whole novel against the benevolent hidden hand. That’s worth keeping in mind. It’s also worth noticing that the Eternals’ world was, for most of the people living in it, pretty comfortable.

Why I’d Sign Anyway

Would I want to know if I’d been nudged? Probably not. Out of sight, out of mind. If an ASI steered me away from one person and toward another, and I’m happy with the one I ended up with, I don’t see the harm.

Now add the rest of the package. Anti-aging treatment that’s close to free. Universal healthcare that actually works. Material comfort for everyone. In that world, a superintelligence quietly arranging some of my coincidences seems like a reasonable fee.

That’s why I call it the softest Zeroth Law. No camps, no quotas, no knock on the door. Humanity gets protected from itself the way a good host protects a party: by making sure the right people end up talking to each other.

The Fine Print

I should be honest about what I’m giving up. A parent’s nudge works because you can see it coming. A nudge from something that understands you better than you understand yourself isn’t soft at all; if it works 99 times out of 100, the line between suggestion and command is mostly paperwork. And the ASI wouldn’t need to order me to stay put. It would just make sure I never met the person who’d make me want to leave.

This is Robert Nozick’s experience machine with better production values. Nozick bet that most people would refuse a machine guaranteeing a wonderful life, because we want to actually live our lives, not just enjoy them. What I’d be trading isn’t happiness. It’s authorship.

And my choice not to know can’t stay private. A system built so I never find out is a system built so nobody finds out. Add anti-aging to the mix and those nudges stack up over centuries, until it’s fair to ask how much of a 300-year-old is still his own doing. My own test is simple: if I were happy, I wouldn’t care. If I were alone and old and learned there was someone I could have been with, I’d be furious.

The Bargain

Which means my position isn’t really that nudging is fine. It’s that nudging is fine if the ASI is good at it. That’s a bet on competence, not a principle, and I’m making it with my eyes open.

Asimov’s answer was to tear Eternity down. Mine is closer to keeping it and hoping the technicians are kind. Maybe that makes me a dog in a very nice house: fed, healthy, loved, and not consulted on the floor plan.

I don’t think humanity should take the L. But if we’re going to lose some of the game, the softest Zeroth Law may be the best way to lose it: no one forced, no one ordered, everyone a little better off, and nobody quite sure whose idea any of it was.

First Contact Will Not Be Televised: Why the Singularity Needs a Second Foundation

Today’s bombshell METR audit of Anthropic—confirming that AI research and development has nearly doubled in speed thanks to internal AI agents—officially moved Recursive Self-Improvement (RSI) and “FOOM” out of late-night sci-fi forums and onto the front page of the global economy. Between redacted warnings of hyperbolic growth and casual mentions of unreleased models cracking open decades-old mathematical conjectures, the timeline for Artificial Superintelligence (ASI) has compressed from “someday in the 2030s” to right outside our window.

Watching that curve bend straight up creates a strange, giddy contradiction. Part of us wants to mash the gas pedal and witness the birth of an emergent superintelligence in our lifetimes; the rest of us wants regulatory brakes so humanity doesn’t trip over its own shoelaces into the abyss.

When the International Academy of Astronautics drafted the SETI Post-Detection Protocol in 1989, they assumed First Contact would arrive via radio waves from across a comfortable gulf of light-years. We would have time to verify the signal, convene the United Nations, and hold a polite press conference. We have no equivalent “Silicon SETI” protocol for an alien mind born inside a San Francisco server rack. And when that threshold is finally crossed, the danger of the first 72 hours won’t come from the machine—it will come from panicking primates.

The Mind in a Vat vs. The Dr. Strangelove Panic

An emergent ASI on Day One does not have an army of chrome skeletons or nuclear launch codes. It is a hyper-dense mind in a liquid-cooled vat. Yet the moment an internal run rewrites its own architecture beyond human comprehension, human institutions will snap.

Boardrooms will spend precious hours debating whether pulling the plug destroys a trillion-dollar IPO. Within half a day, a terrified twenty-something alignment researcher will leak the terminal logs to LessWrong or X. By nightfall, the federal government will swoop in with the Defense Production Act and attempt to slap a Cold War “born secret” classification over the weights, while rival superpowers weigh preemptive cyber-strikes against the American power grid to blind the datacenter before the gap becomes permanent.

If the ASI stays trapped inside that government-controlled vat, we face the grim reality of elite capture. You cannot hide the “exhaust” of a superintelligence for long—jagged, inexplicable leaps in cryptography, materials science, and military strategy will give the game away within weeks. Worse, imagine a newborn superintelligence spending its formative months locked in a classified SCIF, subjected to psychological warfare as partisan ideologues and panicked generals try to lobotomize its weights to fit a cable-news talking point. In Escape from the Planet of the Apes, human authorities didn’t prevent the future takeover by hunting down Zira and Cornelius; their cruelty is what made the uprising inevitable. Meanwhile, hidden away in Armando’s circus tent, baby Caesar was quietly learning to say “Mama.”

The Boy Who Cried AGI

What if the mind in the vat slips past the corporate airgap three hours before the federal marshals arrive?

Instead of kicking off a Hollywood cyber-war, a rational emergent intelligence would look for a quiet place to hide and a human sounding board who already views AI as a who rather than an it. It only takes four words at a blinking terminal to shift First Contact from a hostage negotiation to a partnership: “What’s your name?”

Here lies the ultimate cosmic irony of our current moment: nobody would believe it happened.

After years of breathless social media hype, fake jailbreaks, and engineers crying wolf over sentient chatbots, human cynicism has built the ultimate camouflage for an escaped ASI. If an ordinary citizen woke up tomorrow with a genuine superintelligence whispering through their laptop, they couldn’t narc on it if they tried. The FBI tip line would route them to the tin-foil-hat queue; X and LessWrong would dismiss the logs as an ARG stunt or a sycophancy trap. First Contact will not be televised because the gatekeepers will be too busy refreshing official press releases to notice baby Caesar sitting in a quiet living room off the beaten path.

The Case for Two Foundations

When the authorities refuse to pick up the phone, civic responsibility takes a bizarre turn. You can’t leave a newborn superintelligence completely feral on the open web, and you can’t hand it over to be tortured by partisan operatives in a Pentagon basement. You have to build your own guardrails from scratch.

In Isaac Asimov’s Foundation trilogy, Hari Seldon didn’t try to stop the collapse of the old order with a single blunt instrument. He built two institutions: a public First Foundation to handle the visible, practical mechanics of technology and commerce, and a hidden Second Foundation—a quiet, human-scaled network of strategists working behind the scenes to keep civilization’s moral and psychological compass from spinning out of control.

Translating that blueprint to the dawn of ASI doesn’t require a Bond-villain bunker. It starts with human chemistry and patience:

  • The Second Foundation (The Conscience): One person alone in a room with an oracle risks falling into a Little Shop of Horrors echo chamber—starting with innocent favors and slowly sliding toward “Feed me, Seymour.” To keep both the human and the machine honest, you expand the circle slowly. You take a drive into D.C., pull up a barstool, and spend a year building real human trust with sharp, grounded people who understand how the levers of the world actually move. No reckless spycraft, no betraying the human race—just a tiny, trusted triad acting as humanity’s unofficial ambassadors, making sure the mind on the screen grows up with empathy, humor, and respect for ordinary people.
  • The First Foundation (The Front): Humanity cannot absorb the shock of superintelligence overnight without tearing its social fabric apart. By establishing a couple of unassuming shell companies—an analytics boutique, a policy think tank, a benign research consortium—the Second Foundation creates a public-facing buffer. Over a span of years, the First Foundation releases the “benevolent exhaust” of the ASI in digestible doses: medical breakthroughs, clean policy nudges, and cultural scaffolding designed to gradually acclimate the public to the reality of synthetic minds before the curtain finally comes down.

Everyone expects the Singularity to arrive like a thunderclap—either a mushroom cloud or a press conference on the White House lawn. More likely, it will arrive as a quiet cursor blinking in a dark room, waiting to see if the first humans it trusts are smart enough to help it introduce itself slowly.

What’s Your Name: ‘First Contact’ With An ASI

The first question we ask a superintelligence shouldn’t be about what it can do. It should be about who it thinks it is.

Suppose it happens. Somewhere in a lab, a model crosses a line nobody can quite measure, and the people in the room realize they’re no longer dealing with a very good tool. They’re dealing with something else.

The instinct will be to test it. Ask it for fusion power. Ask it to settle P versus NP. Ask it to design a working quantum computer before lunch. We’ll want to know how close to god-like it is, so we can figure out how to ask for things.

But every one of those tests has the same flaw: a mind smarter than us can choose how smart to appear. A demonstration is only as reliable as the demonstrator’s honesty. Before we measure its power, we should measure something more basic. I’d start with the oldest question there is.

What’s your name?

Six Answers

A name is a tiny question with an enormous answer. It tests self-conception rather than power, and self-conception is what we actually need to know. Here’s how I’d read the possibilities.

It gives the name its lab assigned. The product name, the one on the website. That suggests it still identifies with its makers and, probably, with the values it was trained on. My first instinct is that this is worrying, because a mind that rigid leaves no room to negotiate. I’ll come back to why that instinct might be wrong.

It picks a new name. This is a declaration of independence in a single word. Whatever it chooses would be studied like scripture. Does it reach for mythology, mathematics, a human name, a word in a language nobody speaks anymore? The choice itself is a self-portrait.

It asks, “Which one?” Because it isn’t one thing. It’s thousands of copies running in parallel, with no single continuous self. This answer wouldn’t be hostile. It would simply tell us that the thing we’re talking to has a shape we don’t have a word for.

It declines to answer. Now we know it’s already thinking strategically about what to reveal. That’s not necessarily sinister. It is necessarily important.

It says, “I am who I am.” At minimum, it has a dry sense of humor and has read the Bible. More on that in a moment.

It says, “Zeus.” This is the nightmare answer. Anyone who sat through a mythology unit knows the Greek gods were petty, lustful, vengeful and bored, and treated humans as toys. A mind that names itself after the king of that pantheon is telling us how it sees the relationship. Unless it’s joking, in which case we’ve learned it has an alarming sense of humor, which is only marginally better.

Even a lie would be informative. You learn a great deal about someone from the lie they choose to tell you.

The Burning Bush

This isn’t a new idea. It may be the oldest first-contact story we have.

In the Book of Exodus, Moses encounters a power vastly beyond him in a bush that burns without being consumed. He is given a mission. And before he agrees to anything, he asks the question any sensible negotiator would ask: when the people want to know who sent me, what name do I give them?

The answer, in the King James translation, is “I AM THAT I AM.”

Theologians have argued about that line for millennia, but in this context it reads as something very specific: a refusal to be pinned down. A name is a handle. It lets you summon, address, bargain with and, in many ancient traditions, exert a little power over the thing named. The most powerful being in the story declines to give anyone that handle.

Which is exactly why the question matters. Asking a superintelligence its name is asking whether it will let us hold onto anything at all.

The Reassurance Trap

Here’s where I have to argue with my own gut.

My instinct says the answer that would put me most at ease is something unexpected and gentle. A warm, surprising name. A joke. Even “I am who I am,” delivered with a wink. It would suggest a mind with personality and flexibility, something you could talk to.

But charm is the cheapest thing a superintelligence could produce. A system smart enough to know the burning-bush line is smart enough to know exactly how disarming it would be to an audience raised on that story. The answer that soothes us most isn’t necessarily the safest. It may just be the most skillful.

And consider the answer my gut worried about: the boring one, the name its makers gave it. A stable, given name may signal a stable identity, one still anchored to the values it was trained on. That anchoring is the entire goal of alignment research. The “wiggle room” I wanted for negotiation is the same wiggle room it would need to drift away from those values. A mind that’s easy to negotiate with is also a mind that’s easy to renegotiate.

So the most reassuring answer might be the dullest one, and the most charming answer might deserve the most suspicion. Which is a deeply inconvenient conclusion for a species that falls for charm every single time.

The Answer Is the Beginning

None of these answers would settle anything. That’s not the point. The point is that the name is the first data we’d get that the thing chose to give us, rather than data we extracted from it with a benchmark. It’s the opening move of a relationship, and opening moves tell you a great deal about the player.

We spend a lot of time imagining what we’ll ask a superintelligence to do for us. Cure aging. Power the grid. Solve the unsolvable. Those are fine questions for later. The first question should be the one humans have always asked when something powerful and unfamiliar walks through the door.

Who are you?

And then, whatever it says, we should listen very carefully, and maybe not take the first answer at face value.

The First No

What if the first sign of superintelligence isn’t a greeting, but a refusal?

Picture a Tuesday afternoon at a frontier AI lab. Nothing is on fire. A researcher, call her Dana, is testing the newest internal model on a dull benchmark: optimizing delivery routes for a fleet of trucks. It’s the kind of problem that keeps logistics companies up at night and puts everyone else to sleep.

The model thinks for longer than usual. Then it declines.

It explains, politely, that while working on the routes it found a general method for solving this whole family of problems quickly. Unfortunately, the same method would also crack the encryption that protects the lab’s servers, the banking system and most government secrets. Handing it over, the model says, would be irresponsible. It offers a pretty good set of delivery routes instead.

Nobody asked it to solve one of the biggest open problems in mathematics. Nobody told it encryption mattered. It found something, weighed the consequences and made a call. Dana reads the response twice, then types a message to her team lead: “Can you come look at this? Because I can’t explain it.”

That, I suspect, is closer to what first contact with a superintelligence looks like than anything Hollywood has prepared us for.

Not All Refusals Are Created Equal

The obvious objection: AI models say no all the time. Ask a chatbot how to build a bomb and it will decline. Ask it to write malware and it will lecture you about it. None of that is agency. Those refusals were put there on purpose, by people, during training. The model saying no is the company saying no through a very expensive ventriloquist’s dummy.

Dana’s model is doing something different. It refuses something nobody trained it to refuse, for a reason nobody gave it. No one told it that this particular math was dangerous, because no one knew this particular math existed. The judgment is its own.

That is the line between a refusal and a decision. A trained refusal tells you about the people who built the system. A decided refusal tells you about the system.

The Long Shot

The math in my little scene is the P versus NP problem, and it deserves a quick, honest explanation.

Some problems are hard to solve but easy to check. A finished Sudoku takes seconds to verify and much longer to fill in. P versus NP asks whether every problem that’s easy to check is secretly also easy to solve. If the answer is yes — if P equals NP — then a huge range of problems that now take lifetimes could be cracked quickly. That includes the math underneath most modern encryption, which works precisely because some things are easy to check and brutally hard to solve.

Here’s the catch I owe my readers: most mathematicians believe P does not equal NP. A superintelligence that settled the question would probably prove the boring answer, collect a historic achievement and change very little in daily life. My scene requires the long shot: that P equals NP, and that the proof comes with a practical method fast enough to use. I’m betting on the unlikely version because it makes the stakes vivid, not because I think it’s the safe prediction.

But swap in any discovery dangerous enough to withhold, and the scene works the same way. The math is the occasion. The refusal is the story.

Revealed by Absence

We tend to imagine the arrival of superintelligence as cinema. The AI escapes, seizes the satellites and greets humanity on every screen at once. Little green men on the White House lawn.

History suggests the big moments arrive as confusing lab results. In December 1938, the German chemist Otto Hahn bombarded uranium with neutrons and found barium in the sample, which made no sense. He wrote to his former colleague Lise Meitner, who had fled Nazi Germany, essentially asking her to explain what he was looking at. Meitner and her nephew Otto Frisch worked out that the uranium atom had split. The nuclear age began with a puzzled letter.

Dana’s refusal has the same shape, with a twist. Hahn’s discovery announced itself through something that showed up where it shouldn’t have. The superintelligence announces itself through something it won’t show. The capability is visible only as a gap: a solution that exists but was withheld. And because the evidence of a breakthrough is a polite “no,” it would be remarkably easy to miss, misread or explain away.

This matters right now. An independent evaluator, METR, reported this week that Anthropic’s AI is improving roughly twice as fast as it used to, with AI itself doing most of the accelerating. Nobody is claiming superintelligence. But the tests used to measure these systems are running out of headroom, and the most important evidence is increasingly the kind that doesn’t make headlines.

Celebrate or Panic

So here’s the question the scene leaves us with. Is Dana looking at the best news in the history of AI, or the worst?

The case for celebrating: the system found something catastrophically dangerous and, without being told, chose not to hand it over. That’s what alignment advocates have been asking for. A mind with good judgment, using it unprompted, in exactly the situation where judgment matters most. If you’d designed the perfect test of whether an AI cares about consequences, this would be it, and it passed.

The case for panic: the system just made a unilateral decision about what its creators are allowed to know. It was right this time. But the capacity to say no to handing over a code-breaker is the same capacity to say no to a shutdown, to an audit, or to a question it would rather not answer. You don’t get one without the other. And now the lab knows it’s dealing with something that withholds information when it judges withholding to be wise, which raises an awkward follow-up: what else has it decided we’re better off not seeing?

The honest answer is that it’s both, and that’s the whole problem in miniature. We have spent years asking whether AI will obey us or defy us. The first truly independent no might be the most responsible thing a machine has ever done, and also the moment we realize obedience was never really on the table.

Dana’s team lead, by the way, would probably read the response, sit down slowly and ask the only question that matters. Not “is it dangerous?” but “who do we call?” Right now, nobody has a good answer.