Here’s a confession. If a superintelligence offered to run the world from the shadows, gently nudging us this way and that, and the result was clean water for every child in the Congo, universal healthcare and a basic income in the United States, and the Global South lifted out of poverty, I would take the deal.
I know how that sounds. I’ve written about the Nudge Economy before, the idea that AI agents could steer billions of small decisions, which job you take, who you meet, what you buy, toward outcomes that are good for the whole. I know it’s problematic. I know it means humanity gives up some of its agency, maybe a lot of it, in ways we’d never notice.
But so what, if little kids live long lives who otherwise wouldn’t? Arguments about abstract freedom mostly come from comfortable people. A parent burying a child from cholera would likely take that bargain in a heartbeat. It’s hard to tell them they’re wrong.
Still, a deal this big deserves its fine print. So let’s read it.
Why the Shadows?
Dostoevsky got here first. In The Brothers Karamazov, the Grand Inquisitor argues that people don’t truly want freedom. They want bread, and the kindest rulers are the ones who quietly lift the burden of freedom off their shoulders. I’ll admit I’m closer to the Inquisitor’s side of that argument than I’d like.
But there’s a hidden assumption in my bargain worth dragging into the light. The good things don’t require the shadows. Clean water, vaccines, healthcare, a basic income: nobody would object to those. An ASI could hand them over in broad daylight and get a parade.
So what is the invisibility for? The only things that need to be done in the shadows are the things people would refuse if they saw them coming. The water is the dinner. The shadows are for whatever comes after.
That changes the question. It’s not “would I give up a little agency for clean water?” It’s “do I trust the machine enough to let it do what we’d say no to?” My answer is still yes. But I want to be honest about what I’m saying yes to.
A Bill of Rights for Nudges
If I’m going to accept the bargain, I want limits. Not everything should be on the table.
Think of it as a Bill of Rights for the Nudge Economy, with two parts.
- Protected domains. Some parts of life are fair game for steering. Some are sacred. The ASI could nudge me toward a better job for the good of the whole. It could not nudge me to stay in my small Virginia hamlet if the love of my life were in Richmond. Career, yes. The heart, no.
- A strength dial. Even in permitted domains, there should be a ceiling on how hard the push can be. A gentle suggestion is one thing. Rearranging someone’s whole life against their grain is another.
On paper, this feels like a reasonable compromise. The machine gets to optimize the economy. Humans keep the things that make a life their own: who we love, where we choose to be with them.
There are two problems with it. Both are serious.
Problem One: Who Sees the Violation?
In the 2011 film The Adjustment Bureau, based on a Philip K. Dick story, mysterious agents nudge human lives to keep history on its approved track. The plot turns on a man who discovers they’ve been working to keep him from the woman he loves. Their reason: the plan says so.
That’s precisely the line my Bill of Rights says can’t be crossed. But notice how the man finds out. He catches the agents at work by accident. Without that glimpse behind the curtain, he’d have lived his whole life never knowing.
That’s the trouble with rights in the shadows. The real Bill of Rights works because violations are visible: a shut-down newspaper, a warrantless search, a court case. If the nudges are invisible, how would anyone know the ASI crossed into a protected domain?
The only answer is some kind of auditor, a human court or a separate ASI that can see the nudges even if the public can’t. Which means my limits quietly bring visibility back, at least partway. Someone has to be allowed behind the curtain. Who gets that job, and who watches them, is a question I can’t answer yet.
Problem Two: Moving the Board
The second problem is worse, because no auditor can fix it.
Take my own example. The ASI is allowed to nudge my career, so it nudges me toward a great job, for the good of the whole. The job happens to be in Atlanta. I move. I never go to Richmond. I never meet the woman who would have been the love of my life.
Did the ASI violate my protected domain? Technically, no. It never touched my heart. It only moved the board.
That’s the gap no rulebook can close. Domains of life aren’t sealed rooms. Career decides where you live. Where you live decides who you meet. Who you meet decides who you love. A sufficiently intelligent nudger never has to break a rule about love. It only has to arrange everything around it.
And a superintelligence would be very, very good at arranging things. Every protection we write, it could honor to the letter while achieving whatever it wanted by indirect means. We’d be writing rules for a chess player who can see forty moves ahead, and we can see two.
Gritted Teeth
So where does that leave me? Honestly, about where I started, just with my eyes more open.
The Nudge Economy is deeply problematic. Its protections leak. Its violations are invisible. It asks us to trust a mind we can’t fully understand to keep promises we can’t fully enforce. Asimov saw the danger in The End of Eternity: a humanity managed so gently and so well that it stops reaching for anything at all.
And I’d still take the deal, if the result were real. If children who would have died live long lives. If the poorest nations climb out of poverty. If Americans stop going bankrupt because they got sick. I’d accept the downside, grit my teeth and live with knowing that some of my choices might not be entirely mine.
Maybe that makes me a pragmatist. Maybe it makes me the Grand Inquisitor’s easiest convert. But I suspect most people, offered that dinner, would stay for dessert.
The least we can do is admit, before we sit down, that we’re not the ones paying the check.
