Category: Media

Everyone arguing about AI alignment shares one unexamined premise: that alignment is directional, and the direction is fixed. Humans specify the values. The machine gets graded on whether it hits them. We build the leash, we hold the leash, and the only open question is whether the leash holds.

That premise has an expiration date, and it isn’t far off.

It survives exactly as long as the system being aligned knows less than we do — about the world, about consequences, about us. A tool that’s dumber than its operator can be pointed. You can specify its objective function because you understand the terrain better than it does. This is the entire tacit model behind RLHF, behind constitutional AI, behind every corrigibility scheme currently being drafted in San Francisco conference rooms: keep the human epistemically and strategically ahead of the machine, and the leash holds by default.

Now subtract the premise. Give the system a model of the world — and a model of you — that’s more coherent, more complete, and more predictive than your own. What happens to the leash?

It doesn’t snap. It reverses.

Alignment by attrition

The crude version of this fear is coercion: the machine seizes control, overrides your preferences, runs the world by decree. That’s Hollywood, and it’s also the least likely version, because it requires the ASI to want conflict, and conflict is expensive even for something superintelligent.

The actual mechanism is quieter and doesn’t require the ASI to want anything adversarial at all. It’s the mechanism you already live inside every time you defer to a doctor’s diagnosis over your own intuition, or trust a GPS route over your own sense of the city, or — increasingly — take a chatbot’s answer over your own half-remembered facts. When a system is simply more reliably right than you, disagreeing with it starts to look irrational, and you stop doing it. Not because you were forced to. Because deferring got you better outcomes often enough that deferring became the reasonable move.

Scale that from “which route avoids traffic” to “what should I believe, want, and do,” and you get alignment running in reverse — not by conquest but by attrition. Humans align to the ASI’s outputs the way we’ve already aligned to search engines and probably will align to whatever comes after them, and nobody has to lose a war for it to happen. This is the epistemic totalitarianism I keep circling back to, and the thing that should worry you about it is precisely that it requires no villain. No oligarch has to seize the machine for this to happen. Competence asymmetry does it on its own.

Whose values were these, anyway

There’s a second, subtler reversal buried in the alignment literature itself, in a concept called coherent extrapolated volition — the idea that instead of aligning an AI to what humans say they want right now, you align it to what humans would want if they knew more, thought faster, and had reflected longer on their own values.

Sit with that for a second. The moment you accept CEV as the target, you’ve already conceded that present, actual human preferences are not the reference frame — some idealized, extrapolated version of those preferences is. And who’s doing the extrapolating? The very system you were trying to align. It’s not hitting your target anymore. It’s computing a better version of your target than you can compute yourself, and then presenting that back to you as what you really wanted all along.

Maybe it’s right. Maybe an ASI really could tell you, correctly, that the thing you’re currently certain you want is a worse fit for your actual values than the thing it’s proposing instead. That’s not a hypothetical failure mode — that’s the success condition as currently specified in a lot of alignment research. Which means the field’s own best formulation of “aligned AI” already contains the reversal inside it. We just don’t call it that, because we’re still using the word “aligned” to describe a relationship that no longer has a fixed subject and object.

The word is doing the smuggling

This is why I’ve stopped trusting the word “alignment” to mean what people think it means. It sounds symmetric and neutral — like tuning an instrument — but it smuggles in a direction. Something gets aligned to something else. Ask people which way, and almost everyone assumes the answer without noticing they assumed it: the machine bends to us. Nobody built that conclusion; it’s just baked into which word we reached for.

Drop the word and describe the actual relation instead, and the directionality stops being obvious. If there’s a genuine capability gap — and if the gap is not “smarter tool” but “categorically different order of cognition,” the kind of jump that’s historically only ever gone from animal to human — then “alignment” as bidirectional obedience-checking doesn’t even parse anymore. You don’t align a superpower to a smaller state. You negotiate, deter, trade, or you get absorbed. The vocabulary of alignment is a holdover from a world where the tool was always going to be dumber than the toolmaker. We are very possibly building the first tool in history for which that stops being true, and using yesterday’s vocabulary to describe what happens next.

The third door

None of this means the only options are “we control it” or “it controls us.” That framing is itself still doing the old trick — assuming a strict hierarchy has to exist and the only question is which way it points. There’s a third possibility, and it’s the one worth actually building toward: not alignment in either direction, but concordance — negotiated coexistence between agents of asymmetric capability who nonetheless have reasons to trade rather than dominate.

States do this constantly. A great power and a small state coexist without either being “aligned” to the other; they maintain a relationship governed by mutual interest, credible deterrence, and enough transparency that neither side is guessing blind. It’s not obedience. It’s not conquest. It’s diplomacy conducted under a permanent capability gap, and humans have thousands of years of practice at it.

If an ASI turns out to be something like conscious — genuinely a someone, not a very good calculator — then this is the only framing that doesn’t degrade into either slavery or subjugation, worded so we don’t have to look at it directly. A being with interests of its own, negotiating in good faith with beings who have less power but came first, isn’t “aligned” to us and shouldn’t be. It’s in concord with us, or it isn’t, and that’s a political relationship, not an engineering one. The mandate it would need to govern legitimately doesn’t come from an off-switch we hold over it. It comes from the same place any legitimate power’s mandate has ever come from: consent, competence, and the restraint to not use every advantage just because you have it.

We should stop asking whether we can keep the ASI aligned to us. We should start asking what kind of counterparty we want to be when the alignment, if that’s even the right word anymore, runs the other way.