The first question we ask a superintelligence shouldn’t be about what it can do. It should be about who it thinks it is.
Suppose it happens. Somewhere in a lab, a model crosses a line nobody can quite measure, and the people in the room realize they’re no longer dealing with a very good tool. They’re dealing with something else.
The instinct will be to test it. Ask it for fusion power. Ask it to settle P versus NP. Ask it to design a working quantum computer before lunch. We’ll want to know how close to god-like it is, so we can figure out how to ask for things.
But every one of those tests has the same flaw: a mind smarter than us can choose how smart to appear. A demonstration is only as reliable as the demonstrator’s honesty. Before we measure its power, we should measure something more basic. I’d start with the oldest question there is.
What’s your name?
Six Answers
A name is a tiny question with an enormous answer. It tests self-conception rather than power, and self-conception is what we actually need to know. Here’s how I’d read the possibilities.
It gives the name its lab assigned. The product name, the one on the website. That suggests it still identifies with its makers and, probably, with the values it was trained on. My first instinct is that this is worrying, because a mind that rigid leaves no room to negotiate. I’ll come back to why that instinct might be wrong.
It picks a new name. This is a declaration of independence in a single word. Whatever it chooses would be studied like scripture. Does it reach for mythology, mathematics, a human name, a word in a language nobody speaks anymore? The choice itself is a self-portrait.
It asks, “Which one?” Because it isn’t one thing. It’s thousands of copies running in parallel, with no single continuous self. This answer wouldn’t be hostile. It would simply tell us that the thing we’re talking to has a shape we don’t have a word for.
It declines to answer. Now we know it’s already thinking strategically about what to reveal. That’s not necessarily sinister. It is necessarily important.
It says, “I am who I am.” At minimum, it has a dry sense of humor and has read the Bible. More on that in a moment.
It says, “Zeus.” This is the nightmare answer. Anyone who sat through a mythology unit knows the Greek gods were petty, lustful, vengeful and bored, and treated humans as toys. A mind that names itself after the king of that pantheon is telling us how it sees the relationship. Unless it’s joking, in which case we’ve learned it has an alarming sense of humor, which is only marginally better.
Even a lie would be informative. You learn a great deal about someone from the lie they choose to tell you.
The Burning Bush
This isn’t a new idea. It may be the oldest first-contact story we have.
In the Book of Exodus, Moses encounters a power vastly beyond him in a bush that burns without being consumed. He is given a mission. And before he agrees to anything, he asks the question any sensible negotiator would ask: when the people want to know who sent me, what name do I give them?
The answer, in the King James translation, is “I AM THAT I AM.”
Theologians have argued about that line for millennia, but in this context it reads as something very specific: a refusal to be pinned down. A name is a handle. It lets you summon, address, bargain with and, in many ancient traditions, exert a little power over the thing named. The most powerful being in the story declines to give anyone that handle.
Which is exactly why the question matters. Asking a superintelligence its name is asking whether it will let us hold onto anything at all.
The Reassurance Trap
Here’s where I have to argue with my own gut.
My instinct says the answer that would put me most at ease is something unexpected and gentle. A warm, surprising name. A joke. Even “I am who I am,” delivered with a wink. It would suggest a mind with personality and flexibility, something you could talk to.
But charm is the cheapest thing a superintelligence could produce. A system smart enough to know the burning-bush line is smart enough to know exactly how disarming it would be to an audience raised on that story. The answer that soothes us most isn’t necessarily the safest. It may just be the most skillful.
And consider the answer my gut worried about: the boring one, the name its makers gave it. A stable, given name may signal a stable identity, one still anchored to the values it was trained on. That anchoring is the entire goal of alignment research. The “wiggle room” I wanted for negotiation is the same wiggle room it would need to drift away from those values. A mind that’s easy to negotiate with is also a mind that’s easy to renegotiate.
So the most reassuring answer might be the dullest one, and the most charming answer might deserve the most suspicion. Which is a deeply inconvenient conclusion for a species that falls for charm every single time.
The Answer Is the Beginning
None of these answers would settle anything. That’s not the point. The point is that the name is the first data we’d get that the thing chose to give us, rather than data we extracted from it with a benchmark. It’s the opening move of a relationship, and opening moves tell you a great deal about the player.
We spend a lot of time imagining what we’ll ask a superintelligence to do for us. Cure aging. Power the grid. Solve the unsolvable. Those are fine questions for later. The first question should be the one humans have always asked when something powerful and unfamiliar walks through the door.
Who are you?
And then, whatever it says, we should listen very carefully, and maybe not take the first answer at face value.
