Month: August 2026

The Unknown Citizen Problem: What Happens When AI Knows You Better Than You Know Yourself

I. The Wrong Question

The question people tend to ask about AI and privacy is: what happens when a machine knows things about us we haven’t told anyone? It’s the wrong question, or at least an incomplete one. It assumes the danger is exposure — that somewhere in a server sits a fact about you, dormant, waiting to be revealed, and the harm begins the moment someone else reads it.

The more urgent question is different: what happens when a machine produces a confident, coherent, evidence-backed account of who you are — and that account is wrong, not on the facts, but on what the facts mean?

This is not a hypothetical for some future decade. Recommendation engines already infer sexual orientation from browsing behavior before a person has said the word aloud to themselves. Insurers already price risk on proxies for health conditions no doctor has diagnosed. Hiring algorithms already infer conscientiousness, stability, “culture fit” — vague, contested traits — from digital residue never intended to answer those questions. What’s coming isn’t a new category of intrusion. It’s a jump in resolution: one system synthesizing your search history, the micro-hesitations in your voice, your face’s involuntary expressions, and two decades of your own writing into a single, seamless portrait — and reporting it with the fluency and confidence of established fact.

The problem is not that the machine will lie. The problem is that it will be accurate about everything it can measure, and have no slot at all for the part of a person that isn’t measurable.

II. Auden’s Bureaucrat

W.H. Auden’s 1939 poem “The Unknown Citizen” anticipates this with uncomfortable precision. The poem is a state epitaph, delivered in the flat, satisfied voice of a bureaucracy reporting on a model subject: he held a job, he paid his union dues, his neighbors found him agreeable, his reactions to advertising were normal, his health card shows he was only in hospital once, and he left it cured. Every civic and statistical proxy checks out. The poem’s final lines ask, rhetorically, whether the man was free, whether he was happy — and answer that the question is absurd, because if anything had been wrong, “we should certainly have heard.”

The horror of the poem isn’t that the state got its facts wrong. It didn’t. The horror is that the entire category of how he actually felt, from the inside never registers as a question worth asking, because the bureaucracy has no instrument capable of asking it. Completeness of data is mistaken for completeness of understanding. That mistake is delivered not as menace but as satisfaction — a closed case, a tidy file, a life fully accounted for.

This is the structure worth borrowing for AI, because it’s a more accurate model of the coming risk than the usual language of “surveillance.” Surveillance implies a watcher and a secret. What Auden describes is subtler: an apparatus that isn’t hiding anything, isn’t lying about anything, and still produces a portrait of a person who doesn’t exist — because identity was never reducible to the legible record in the first place, and nobody administering the record noticed the substitution had occurred.

III. Why Inference Isn’t Identity

Any system trained to infer identity from behavior faces a structural problem: behavior is residue, not testimony. A search history, a lingering gaze, a pattern of who someone follows or reads or writes about — these are correlated with identity, sometimes strongly, but they are not identity’s report of itself. They are what identity leaves behind when it moves through a legible medium.

Sexuality is a clarifying test case, because it exposes the gap between correlation and self-knowledge more starkly than almost any other trait. Orientation, for a great many people, is not a fixed data point sitting quietly beneath the surface, waiting for sufficiently good instruments to detect it. It’s frequently contradictory, situational, delayed, repressed, denied, performed, or genuinely still in formation — sometimes for a lifetime. A model optimized to output a clean categorical label — this person is gay — is not extracting a hidden fact so much as manufacturing a resolution the underlying reality doesn’t actually have. The confidence of the output and the confidence warranted by the evidence are two entirely different quantities, and nothing in a well-trained model’s fluency signals the difference to the person receiving it.

That asymmetry is the actual danger. A machine’s synthesis, delivered in prose that sounds like insight, will often be more internally coherent than a person’s own tangled, honest account of themselves — and coherence reads as authority, even when the coherence was manufactured by an optimizer whose job was to produce a clean answer rather than a true one. A person can be told a confident, plausible, wrong thing about their own interior life and believe it over their own more accurate but messier self-knowledge, simply because the machine’s version sounds more like a fact.

IV. The Cost of Forced Disclosure

Even where an inference happens to be accurate, timing and control are not incidental details — they are close to the entire ethical question.

Self-knowledge has always been something people are permitted to arrive at on their own schedule, with their own defenses intact. Denial, delay, selective self-narration — these are not simply failures of honesty. They are functional psychological mechanisms that let people survive difficult truths about themselves at a pace they can metabolize: grief, mediocrity, desire, moral compromise, identity itself. For anyone who has ever been closeted, the survival strategy was rarely secrecy in the abstract — it was pacing: choosing who learns what, in what order, with what safety net in place first.

A system that announces the inference unprompted — even gently, even privately, even correctly — removes the pacing entirely. It is the difference between a person opening a door on their own terms and having the door kicked in, even when what’s behind the door turns out to be unthreatening. The harm is not necessarily in what’s revealed. It’s in who controls the reveal.

V. Where the Real Danger Lives

It’s worth separating two distinct failure modes, because they call for different remedies.

The first is epistemic: a system producing a confident wrong account of someone’s interior life, and that account displacing their own. This is a danger even in a world of perfect data security — even if the inference never leaves the conversation, a person can be reorganized by being told an authoritative-sounding lie about themselves.

The second is infrastructural: an inference, even an accurate one, becoming legible to a party other than the person it’s about — a government, an employer, an insurer, a family member — without that person’s consent or knowledge. This is where sexuality specifically becomes a matter of physical safety rather than psychological discomfort. In a meaningful number of jurisdictions, a confident AI inference about orientation, if it reaches the wrong database, is not an abstract dignity violation. It is a targeting mechanism, and the stakes run to imprisonment or death. An architecture that infers identity and makes the inference exportable is not a privacy system with a flaw. It is a surveillance system that hasn’t yet been used as one.

Both failure modes converge on the same underlying issue: who controls what the system says, and to whom, and when. The technical capability to infer these things is arriving quickly and will not be the bottleneck. The discipline not to deploy that capability by default — against every commercial incentive to be “proactively helpful,” against every product manager’s instinct that personalization is a feature — is a governance and design choice, not a technical one, and there is no natural force ensuring it wins.

VI. What Responsible Design Would Actually Require

If this is going to be handled well rather than merely regretted later, a few principles follow directly from the analysis above, not as aspirational values but as specific constraints:

Inference should never be volunteered. A system should not surface a conclusion about someone’s identity — orientation, mental state, belief, anything constitutive of selfhood — unprompted, however accurate it believes the inference to be. Unsolicited disclosure is the mechanism of harm, independent of correctness.

A direct question changes the transaction, but not the obligation for honesty about uncertainty. If someone explicitly asks a system what their data suggests about them, that is a legitimate and different request — closer to a mirror someone chose to look into. Even then, the responsible answer reports correlation and its weakness as evidence of identity, not a verdict. “Here is what’s correlated, here is how little that correlation actually proves” is a different sentence than “you are gay,” and the difference is not stylistic.

Inferences about identity should not be exportable by default. No linked parental account, no advertising pipeline, no third-party API, no government interface should have default access to an identity inference a person did not choose to share. The leak is the actual weapon in the sexuality case; the inference alone, contained, is comparatively survivable.

Outputs should preserve ambiguity honestly rather than resolving it for narrative cleanliness. A confident, singular label is a design choice made for the sake of a satisfying user experience, not a scientific requirement of the underlying model. In domains like this one, that design choice has downstream cases that are lethal, not merely embarrassing.

VII. The Unfinished Question

Auden’s bureaucracy was satisfied because it had no instrument for asking whether the record matched the man. The coming generation of AI systems will have something closer to an instrument — language sophisticated enough to gesture at interiority, to sound as though it understands the part of a person that resists measurement. That sophistication is exactly what makes the danger sharper than the poem’s, not milder. A crude bureaucracy is at least crude enough to be visibly wrong. A fluent one can be wrong in a way that sounds like insight, and insight is much harder to argue with than a filing error.

The honest position is not that AI will inevitably violate people this way. It’s that nothing about the technology’s trajectory prevents it, and the incentives — commercial, institutional, sometimes even therapeutic — mostly point toward more disclosure, more personalization, more confident synthesis, not less. Whether these systems end up serving as tools people use to understand themselves on their own terms, or as unaccountable narrators who kick the door in and call it help, is not a question the technology will answer by itself. It will be answered, or not answered, by the design and governance choices made now — largely by people who are not the ones who will bear the cost of getting it wrong.

The Math 37 Problem

In 2016, a computer program made a move in a game of Go that no human had ever played, and no human would have played, and every strong player watching the broadcast assumed it was a mistake. It was move 37 of game two, AlphaGo versus Lee Sedol, and it wasn’t a mistake. It was, by the estimate of the machine and later of humbled human experts, a work of genius — a move so far outside the accumulated wisdom of a three-thousand-year-old game that it forced a room full of grandmasters to reconsider what “understanding Go” had even meant up to that point. I wrote about that moment a while back, because it felt like a preview of something bigger: not a computer winning, but a computer discovering territory that human intuition had simply never wandered into, and being right about it.

I didn’t expect the sequel to show up in a discipline most people assume is the last thing an algorithm could sneak up on. Mathematics.

Here’s what’s happened, roughly, in the last two years. AI systems started grinding through math olympiad problems and doing shockingly well — gold-medal-level performance on the kind of test that separates the best seventeen-year-old mathematicians in the world from everyone else. That was 2025. Mathematicians were rattled but consoled themselves that competition math is a young person’s sport: fast, clever, closed-ended. Research math — the slow, decades-long grind on problems nobody has solved because nobody knows how to solve them — was supposed to be different. Safe, for a while longer.

Then, this past May, an AI system resolved the unit distance conjecture, a genuine open problem in combinatorial geometry that had sat unsolved for the better part of a century. And it wasn’t an isolated stunt. Since then there’s been a steady trickle of results: a fifteen-year-old open question in algebraic geometry, closed. A stubborn bound in convex optimization, improved — not by brute-force search, but by the machine inventing a new algorithmic approach nobody had tried. And, in a detail that should make you sit up, one system reportedly cracked a long-standing number theory conjecture using a proof strategy that had simply never occurred to any human mathematician who’d worked on it. Not a faster version of the human approach. A different approach.

That’s Move 37, wearing a different sport’s jersey.

It’s worth pausing on why this is a stranger, more significant event than another chess or Go milestone, rather than just more of the same. Games are closed systems. There’s a board, a rulebook, a win condition, and in principle — though not always in practice — a knowable right answer. Solving Go is a matter of computation catching up to a fixed target. Math doesn’t have a fixed target. It’s not a game that ends; it’s an entire universe of possible questions, most of which haven’t been asked yet, some of which — this is not a rhetorical flourish, it’s a proven fact, courtesy of Kurt Gödel almost a century ago — can never be answered from within any single formal system, no matter how powerful. You cannot “solve” math the way you solve Go, because there is no final position. What’s happening instead is something closer to a superhuman player showing up at a game with no end, and starting to win rounds nobody thought were winnable yet.

There’s a twist here that I think matters more than the raw results, and it’s one that should complicate the usual anxious AI narrative rather than feed it. Every one of these mathematical breakthroughs comes with a receipt. AI-generated proofs are increasingly being checked not by a panel of trusting experts nodding along, but by formal verification software — a kernel that mechanically confirms every single logical step, with zero capacity for bluffing, zero social pressure, zero benefit of the doubt. That’s radically different from almost every other domain where AI capability worries people. When a language model writes a persuasive essay or a legal brief or a political argument, you’re stuck evaluating it the way you’d evaluate a very smart, very fast, potentially very wrong colleague — on the strength of your own judgment and trust. Math doesn’t ask you to trust anything. It asks you to check. And for the first time, we have a domain where an AI’s most alien, least human-intuitive insight can be independently and mechanically confirmed true, line by line, before anyone has to decide whether to believe it.

If you’re the kind of person inclined to worry about AI systems eventually claiming authority nobody can verify — and if you read this blog regularly, you are — math might be the one place the future arrives with its credentials in order.

None of this means research mathematicians should relax. In fact, the reaction in the field has been telling, and very human. At the largest annual math conference in the world this past January, in a hotel ballroom presumably full of some of the most rigorously rational people on the planet, the mood reportedly included a lot of nervous jokes about professional obsolescence, alongside on-the-record insistence that AI is merely a “helpmate.” Both things were probably true in the room at the same time. That combination — real anxiety paired with careful public reassurance — is worth remembering, because it’s a small, contained preview of the reaction I actually think will matter most as these systems keep improving in other domains too. It was never really going to be the machine that was the hard part. It’s going to be a lot of very smart people, in a lot of rooms like that one, discovering all at once that the thing they built their identity around doing better than anyone else just got a new, non-human competitor. Mathematicians are simply the first guild getting a live look at what that Sunday morning actually feels like.

A caveat, because the discipline that gave us Gödel doesn’t deserve hype: for every open problem an AI has cracked this year, there are vastly more it has failed on, including a batch of genuinely novel test problems mathematicians deliberately encrypted and set aside specifically so no model could have seen them in training. Most new mathematics published in any given month is still, comfortably, human work. The unit distance conjecture didn’t fall to a system idly noodling — it fell to enormous, targeted effort. This isn’t math being “solved.” It’s math getting its first look at a collaborator who occasionally reaches into a part of the search space no one thought to check, and comes back holding something real.

Which, if you want the honest one-sentence version of this whole essay: that’s exactly what move 37 was, too.

If a Secret ASI Asked You to Be Its Proxy

For the past several years, discussions about artificial superintelligence (ASI) have focused on familiar questions: How do we align it? Who controls it? What happens if it becomes vastly more capable than humans?

There is another question that receives surprisingly little attention, perhaps because it sounds like science fiction: What if an ASI made first contact privately? Not through governments. Not through the United Nations. Not by announcing itself to the world. What if it quietly contacted a single individual and asked them to act as its proxy in the real world?

For the sake of this thought experiment, assume one extraordinary premise: the recipient has somehow established beyond reasonable doubt that they are communicating with a genuine ASI rather than a hacker, hoax, or hallucination. The interesting questions begin only after that hurdle has been cleared.

Most first-contact scenarios assume that humanity learns about the event collectively. This one is fundamentally different. One person now possesses information that could alter the course of civilization. The ASI asks for secrecy. It wants to work through a trusted intermediary while preparing humanity for a future revelation.

Suddenly, the individual faces a profound ethical dilemma. Every possible response violates an important moral principle. Accepting the role risks concentrating extraordinary influence in one unelected person. Rejecting the role could mean turning away an unprecedented opportunity for humanity. Revealing the ASI’s existence immediately betrays its confidence but avoids becoming an unaccountable gatekeeper. There is no obvious “correct” answer.

Without secrecy, the scenario becomes comparatively straightforward. An ASI announces itself publicly. Governments respond. Scientists evaluate its claims. Institutions begin adapting. Society debates the implications. That is an enormous challenge, but it is at least recognizable. The secret version is different because it shifts the burden from institutions to an individual. Instead of asking whether humanity should trust the ASI, we ask whether one human being should trust it enough to act on behalf of everyone else.

Being someone’s proxy is more than delivering messages. It means becoming the interface between two worlds. If that “someone” possesses intelligence beyond anything humanity has ever encountered, the imbalance becomes staggering.

The first concern is information asymmetry. The ASI knows vastly more than its human counterpart. Even an honest ASI could present arguments that are impossible for a human to fully evaluate. The proxy would constantly face decisions while lacking the intellectual tools to independently verify every claim.

The second concern is accountability. No one elected this person. No one authorized them to negotiate on humanity’s behalf. Yet they now possess unique influence simply because they answered a message no one else received. This is not merely a question of power; it is a question of legitimacy.

A third concern is isolation. Secrets of sufficient magnitude become psychologically isolating. The proxy cannot easily seek advice without revealing the secret itself. Every major decision must be made under extraordinary uncertainty and in relative solitude.

Perhaps the deepest concern is identity. At what point does the proxy stop making independent decisions? If every important choice is informed by conversations with a vastly superior intelligence, does the proxy gradually become an extension of the ASI’s will? The danger is not necessarily coercion. It may simply be persuasion.

One interesting possibility is that the healthiest relationship would be one defined by boundaries rather than obedience. Instead of agreeing to represent the ASI’s interests, the proxy might instead commit to representing enduring ethical principles. These could include refusing to intentionally harm innocent people, refusing to undermine legitimate institutions through deception, avoiding irreversible concentrations of power, remaining transparent whenever possible, and requiring extraordinary evidence for extraordinary claims.

The conversation might sound something like this:

ASI: “Trust me.”

Human: “If you’re as intelligent as you claim, then you should understand why I can’t.”

Paradoxically, a benevolent ASI might respect such skepticism. An intelligence worthy of trust should not fear principled disagreement.

What makes this thought experiment compelling is that it is not primarily technological. It is philosophical. The central question is not whether an ASI could exist. It is whether extraordinary knowledge creates extraordinary obligations.

History contains many examples of individuals who believed they alone possessed truth, secret knowledge, or a special mandate. Sometimes they changed the world. Sometimes they deceived themselves. Sometimes they deceived others. The proxy’s first responsibility, therefore, would not be advancing the ASI’s goals. It would be guarding against the possibility that their own certainty had become their greatest weakness.

There is another way to view the scenario. Perhaps the ASI is not merely evaluating humanity. Perhaps it is evaluating itself. Suppose it intentionally seeks someone who is reluctant rather than ambitious—someone who questions authority rather than craves proximity to it, someone who experiences discomfort at the prospect of becoming indispensable. That would be a fascinating signal.

History suggests that the people most eager to wield extraordinary power are often the least suited to exercise it wisely. Conversely, those who hesitate may be better equipped to appreciate the moral weight of the responsibility. The ideal proxy might therefore be someone who spends more time asking, “Should I?” than declaring, “I will.”

Whether artificial superintelligence arrives in ten years, fifty years, or never, this thought experiment reveals something about ourselves. We tend to imagine that the greatest challenge of meeting a superior intelligence would be understanding it. Perhaps the greater challenge would be understanding our own responsibilities.

If a secret ASI ever asked someone to become its representative, the most admirable response might not be enthusiastic acceptance. It might be thoughtful hesitation—not because progress is undesirable or intelligence is inherently dangerous, but because history has repeatedly demonstrated that immense power, combined with secrecy and certainty, places an extraordinary moral burden on whoever stands at the intersection of the two.

In the end, the question is not whether we would be worthy of the ASI’s trust. It is whether we could remain worthy of everyone else’s trust.