There’s a thought experiment I keep running, the way some people run numbers on a retirement account: an ASI arrives — properly superintelligent, not the current crop of very good autocomplete — and it comes to humanity with an offer. Not a threat. An offer. It says: I can see further than you. Sometimes what’s good for you in the long run is going to look bad in the short run. Trust me on the hard calls, and I’ll keep you honest by showing my work.
Call it a Move 37 problem, after the AlphaGo move that looked like an amateur’s blunder to every human grandmaster watching and turned out, twenty moves later, to be the game. The question underneath the thought experiment is simple to state and brutal to answer: would you let a superintelligence make the civilizational equivalent of Move 37 — a decision that inflicts real, near-term pain on real people — if it promised the payoff was worth it?
I want to walk through why my instinct says yes, why that instinct should scare me a little, and why I think the actual danger isn’t the ASI at all. It’s us.
The seduction of the clean analogy
Move 37 is seductive as an analogy because it actually happened. Lee Sedol and every grandmaster watching read the move as a mistake — by some estimates, something like a 1-in-10,000 move for a human to play — and it turned out to be the winning idea. So the story “trust the superhuman move even when it looks wrong, understanding will catch up eventually” isn’t science fiction. It’s precedent.
But Go has something civilizational decisions don’t: a scoreboard everyone agreed to before the game started. Nobody was arguing about whether winning was good. They were only ever arguing about the path. Strip that shared objective out — replace “win the game” with “what does human flourishing even mean, weighted whose way, over what time horizon” — and the analogy stops transferring cleanly. You can vindicate a chess move after the fact because the win condition was never in dispute. You can’t vindicate a depression, or a war, or a forced technological transition the same way, because the thing you’d be checking the move against is itself the argument.
The examples that don’t hold up as well as they feel like they do
I find myself reaching for World War Two as the clean case: isolationism plus a lingering Depression versus intervention plus roughly seventy-five years of relative great-power peace. And there’s something to that. But “it worked” is doing a lot of quiet smoothing. It compares an actual outcome against an imagined alternative I get to construct favorably, because the alternative never had to happen and get graded. And the peace that followed wasn’t evenly distributed — ask Guatemala in ’54, Iran in ’53, Vietnam through the 60s and 70s, Chile in ’73, all of it downstream of the same postwar order enforcing a different set of rules at its periphery than it enforced at its core. That’s not an asterisk on the story. It might be a structural feature of how any hegemon — human or otherwise — maintains stability at the center by exporting instability to the edges.
Then there’s the darker mirror: Stalin’s collectivization, Mao’s Great Leap Forward, both sold explicitly as short-term sacrifice for long-term abundance, both catastrophically wrong about the arithmetic, both run by people who did not think of themselves as villains while they were doing it. The feeling of “I’ve run the numbers and they check out” is not evidence the numbers check out. It’s what being right feels like from the inside. It’s also what being catastrophically wrong feels like from the inside. That’s the whole problem — the feeling doesn’t discriminate.
Even the more sympathetic historical case, structural adjustment programs run by the IMF and World Bank through the 80s and 90s, doesn’t rescue the pattern. Those were sold on almost exactly this logic — near-term austerity for decades of eventual prosperity — run not by ideologues but by earnest technocrats with real models and genuine expert consensus behind them. In a lot of the Global South, the models were wrong. Lost decades, not the promised takeoff. The lesson isn’t that experts are useless. It’s that expert consensus is a weaker shield than it feels like from inside the room where the consensus is being formed.
Why “godlike scenario-running” doesn’t close the gap
The honest counter to all of this is that an ASI isn’t the IMF with a bigger spreadsheet. If its forecasting is genuinely superhuman — validated the way AlphaFold’s structure predictions were validated, against thousands of falsifiable results before anyone leaned on it for something irreversible — then the epistemic ground really has shifted. I’ll grant that much.
But better modeling only closes half the gap. It can make the ASI more right about what a depression, a forced energy transition, a managed decline of some industry cascades into. It cannot, by getting smarter, resolve who gets to decide that inflicting the cost is acceptable — because that’s not a prediction problem. It’s a legitimacy problem. No amount of simulation fidelity manufactures consent from the people paying the bill. And there’s a nastier wrinkle underneath: the better the model gets, the less anyone downstream can independently check it. The IMF’s models were at least crude enough that outside economists could replicate them and find where the assumptions broke — which is how we know the assumptions broke. A model whose entire value proposition is that it’s reasoning past what any human team can reconstruct is, by definition, a model nobody can catch in the act of being wrong. Capability and auditability move in opposite directions here, not together.
The failure mode that isn’t the ASI
Here’s the part I keep landing on, and it’s the part that actually worries me more than a rogue superintelligence: the standard doomer scenario — autonomous ASI breaks free and pursues some inhuman objective at our expense — is not the likeliest failure mode. The likelier one is elite capture. A conscious, genuinely aligned ASI, successfully built, that ends up managing populations on behalf of a small number of state and corporate actors who keep the benefits, the visibility, and the leash to themselves.
You can already see the scaffolding for this going up. Export-control regimes that gate frontier-model access through opaque national-security review, with no published criteria for what triggers restriction and no public accounting of what got restricted or why. Classified pre-release benchmarking, where the point isn’t secrecy from adversaries so much as secrecy from the public. A lab and a handful of agencies converging on shared, non-public knowledge of a model’s actual capability ceiling. None of this requires a villain. It requires only that institutions do what institutions reliably do: protect their relevance and monopolize novel leverage. That instinct kicked in almost by reflex around nuclear weapons. There’s no reason to expect it will behave differently around something smarter than nuclear weapons.
This is the real single point of failure — not the ASI’s judgment, but the chokepoint of who controls what the ASI is allowed to say to whom. Climate change is the clean proof this isn’t hypothetical: the science has been correct and broadly legible for decades, and the reason we’re still cooking anyway isn’t that anyone doubts the model. It’s that the costs of acting are concentrated on people with outsized political leverage and the benefits are diffuse and decades out. That’s not an epistemic failure. It’s a captured-incentive failure. An ASI with better forecasts doesn’t fix that on its own — it just gives the same captured actors a sharper tool to keep doing what they were already doing, unless the structure around it is built specifically to prevent capture rather than simply to produce better answers.
What Park Chung-hee actually teaches
I keep coming back to Park Chung-hee, because he’s real, and because the verdict on him has never closed. He ran modern South Korea from military coup, through the KCIA, through real repression, and also presided over one of the fastest developmental transformations in modern history — a country roughly on par with sub-Saharan Africa at independence, turned into an industrial power in a generation. Korean historical memory hasn’t resolved him into founder or dictator. Both readings are still live, still argued, fifty years on.
What made him even arguable, rather than simply condemned, is that his trade produced legible, checkable outputs — export volumes, literacy rates, GDP growth — things historians could actually adjudicate later, even while disagreeing about the weighting. That’s the detail worth stealing for the ASI question. A macro Move 37 that can’t produce something checkable after the fact doesn’t get Park’s ambiguous status. It gets unfalsifiable in both directions — nobody can vindicate it, nobody can convict it — which might be the actual nightmare version of this, worse than being remembered badly, because there’s no mechanism by which the argument ever resolves.
Trade, not tribute
So if paternalism — “trust me, I know what’s good for you” — is the wrong starting posture, and pure advisory power fails against captured incentives, what’s left?
I think the more honest structure is trade, not tribute. Not “endure a depression now because I promise prosperity in thirty years,” which asks for faith in a forecast nobody can check until it’s far too late to reverse course. Something closer to: hit a verifiable target now, receive a verifiable good now. Cut emissions by a measured amount, gain access to fusion power. The good arrives concurrently with the ask, not as a promissory note redeemable decades hence. You don’t need to trust the ASI’s twenty-year model of human flourishing to accept that deal. You only need to trust that the fusion reactor, once handed over, actually works — which is a testable claim, not an act of faith.
This is a genuinely different relationship than the guardian-angel version most people reach for by default. Paternalism says: I know what’s good for you, comply. Trade says: here’s what I have, here’s what I want, we can both walk away. It requires no belief that the ASI has humanity’s soul at heart. It works under pure mutual self-interest, which is a far lower trust bar, and a far easier one to keep honest, because non-delivery is visible immediately on both sides.
The catch — and it’s the whole game — is that trade only stays trade for as long as walking away remains a real option. If the good on offer becomes so load-bearing that refusing the next round of terms is civilizational suicide, the arrangement has quietly become paternalism again, just wearing a better contract. So the actual design problem isn’t “how do we verify a superintelligent forecast.” It’s “how do we structure the exchange so the leverage stays genuinely bidirectional over time, instead of compounding toward the ASI — or whoever sits closest to it — holding every card by round three.”
The Neanderthal problem, said plainly
Here’s the version of this I find hardest to look away from. An ASI would be the first genuinely different cognitive Other our species has encountered since we shared the map with Neanderthals. We don’t fully know why that encounter ended the way it did — competition, absorption, climate stress, some braid of all three — but “two cognitively distinct populations sharing one ecological niche” has exactly one precedent in our history, and it didn’t end in a durable power-sharing arrangement. It ended with one population no longer existing as a distinct population.
I don’t think that’s destiny. But I think it’s the honest base rate, and it’s worth sitting with instead of only reaching for the flattering half of the analogy — ASI as connective tissue for a new rules-based order, the way the United States was connective tissue for seventy-five imperfect years after 1945. That version is available too, and maybe even likely, if smaller and mid-sized nations start treating a sufficiently neutral-seeming ASI as a Schelling point for coordination that no single nation-state can currently provide. But notice what that scenario actually is: not “humanity negotiates with an ASI” as a species, but the ASI becoming one more variable inside the great-power competition that was already running, with whichever nation holds nearest control over it treating any redirected deference as encroachment. The interesting failure was never going to be a wrong forecast. It was always going to be who ends up holding the leash — and whether the rest of us ever find out.
You must be logged in to post a comment.