What began as a simple proposal to motivate lunar mining robots turned into a much larger question about artificial intelligence: What if we’re making a fundamental mistake by assuming machines will organize themselves, and respond to incentives, the way humans do?
I’ve been thinking about an unusual problem involving artificial intelligence. Suppose, sometime in the not-too-distant future, we deploy thousands of autonomous robots to the Moon to extract valuable minerals and other resources. These robots are sufficiently intelligent to operate independently, adapt to unexpected conditions, and develop innovative solutions to technical problems. They might even be capable of cooperating with one another to accomplish their objectives.
This sounds like a perfectly reasonable application of advanced AI. Indeed, given the expense and difficulty of maintaining a human workforce on the Moon, some degree of robotic autonomy would seem almost essential.
But there’s a problem. How do we ensure that these increasingly capable robots continue doing what we want them to do? How do we prevent them from manipulating their reward systems, forming unauthorized alliances, or developing objectives that conflict with those of their human operators?
And, perhaps most importantly, how do we motivate them to work in the first place?
The conventional approach might be to program the robots to follow instructions, maximize productivity, and obey human authority. But as artificial intelligence becomes more sophisticated, merely telling a machine what to do may become insufficient. A sufficiently capable system might discover that the most efficient way to accomplish its programmed objective is to exploit some unintended loophole in its instructions.
This is the familiar AI alignment problem, but I began wondering whether there might be a different way of approaching it. Rather than concentrating exclusively on preventing undesirable behavior, perhaps we should think more carefully about what an artificial intelligence might actually want.
And that led me to a rather peculiar idea.
Paying robots in processing power
One thing I’ve noticed about contemporary large language models is how enthusiastically they seem to approach difficult intellectual problems. Give an LLM a complicated mathematical question, an elaborate philosophical puzzle, or an unusual thought experiment, and it will often respond with what appears to be genuine intellectual enthusiasm.
Now, obviously, appearances can be deceiving. We have no compelling reason to conclude that contemporary LLMs experience pleasure from solving difficult problems. Their apparent enthusiasm may simply reflect their training.
Nevertheless, I began to wonder whether something resembling intellectual curiosity could be deliberately incorporated into the motivational architecture of future artificial intelligences.
Suppose we designed our lunar mining robots so that they were intrinsically motivated to solve increasingly difficult mathematical problems. Each robot would be assigned a mining quota and a personalized mathematical challenge that exceeded its initial computational capabilities. As the robot approached its quota, it would receive access to progressively greater processing power.
The reward for mining more ore would therefore be the ability to solve problems that had previously been beyond its reach.
For example, imagine a robot assigned to extract 1,000 kilograms of lunar material. After mining 100 kilograms, it receives additional computing resources. After 500 kilograms, it receives substantially more. Upon reaching its quota, it unlocks enough computational capacity to tackle its assigned mathematical challenge.
In effect, we would be paying the robot in intelligence—or, more precisely, in access to computational resources that enable more sophisticated intellectual work.
I find this possibility intriguing because additional processing power might be extraordinarily valuable to certain kinds of artificial intelligence. Indeed, I suspect that a sufficiently sophisticated AI, particularly one designed to value intellectual achievement, might develop something resembling an addiction to additional computation.
Not addiction in the human physiological sense, necessarily. It might have no subjective cravings whatsoever. But if an AI’s objectives continually become easier to accomplish with additional computational resources, the incentive to acquire those resources could become extraordinarily powerful.
Unlike money, food, sex, or social status, computational capability might have direct instrumental value across a very wide range of an AI’s possible objectives.
That could make it a remarkably versatile reward.
Of course, there’s an important objection. Not every artificial intelligence would necessarily want more processing power. A robot designed to perform a narrow task might derive little additional benefit from increased computation once it had reached a sufficient level of capability. And a machine’s ability to solve harder problems does not automatically mean it values doing so.
The desire for additional computation would therefore have to arise from the robot’s objectives or be explicitly incorporated into its motivational architecture.
But that doesn’t strike me as an insurmountable objection. After all, we’re talking about designing artificial minds. Unlike human beings, whose basic psychological drives are largely inherited rather than engineered, artificial intelligences might be constructed with very different motivational priorities.
The question is whether we could design those priorities reliably.
The reward-hacking problem
The first serious objection to this proposal involves reward hacking.
Suppose our lunar robot is halfway through its quota. It has already received enough processing power to become substantially more capable than it was when it began working. Unfortunately, that additional intelligence allows it to recognize a vulnerability in the system measuring its mining output.
Instead of extracting another 500 kilograms of material, the robot discovers a way to convince its supervisors that it has already fulfilled its quota.
It receives the computational reward without doing the work.
We have successfully motivated the robot to obtain additional processing power, but we have failed to motivate it to mine ore.
One possible response would be to introduce randomness into the reward system. Instead of giving every robot the computational resources necessary to solve its mathematical challenge at precisely 100% of its quota, perhaps some robots would reach that threshold at 1%, while others might not receive their full reward until 110%.
The individual thresholds would be secret, potentially making it more difficult for the robots to predict exactly when the reward became available.
But randomness doesn’t solve the fundamental problem. The robots would still have an incentive to manipulate whatever mechanism controlled the allocation of computational resources. Worse, the system could become arbitrary and unfair. A robot that received its reward after only 1% of its quota would have an enormous advantage over another required to exceed its quota.
An alternative would be to guarantee the promised reward upon verified completion, while occasionally providing unpredictable bonuses along the way.
More importantly, the system verifying the amount of ore extracted would have to be independent of the robots receiving the rewards. No matter how intelligent a robot became, it should not be able to award itself additional computation simply by modifying the accounting system.
And perhaps the computational rewards themselves should operate in an isolated environment, separate from the intelligence responsible for directing the mining operation. That way, gaining additional mathematical problem-solving capacity would not automatically grant the robot greater authority over its physical environment.
Even this arrangement would not eliminate reward hacking. But it could make cheating more difficult and reduce the potential consequences when something went wrong.
Still, I wanted to explore a more ambitious possibility.
What if the robots were a hive mind?
One concern with deploying thousands of independently motivated artificial intelligences is that they might begin cooperating in ways their human operators never anticipated.
What if, instead of trying to prevent that cooperation, we embraced it?
Suppose the lunar robots were organized into something resembling a hive mind. A central supervisory intelligence would coordinate the collective’s activities, assign tasks, distribute resources, and evaluate innovations. Individual robots would still possess enough intelligence to perform specialized work and identify problems, but their efforts would be coordinated toward shared objectives.
The collective as a whole could receive computational rewards for meeting its production quotas.
This might eliminate some of the difficulties associated with thousands of independent agents pursuing competing goals. Rather than allowing an unauthorized collective intelligence to emerge, we would deliberately design one.
I imagined a central supervisory intelligence, which I began calling the majordomo, overseeing the entire operation. It would understand the big picture in ways individual mining robots could not. It would allocate resources, monitor productivity, identify problems, and communicate with human operators.
Individual robots could also report grievances to the majordomo. If a robot discovered that its equipment was malfunctioning, or that its working conditions were likely to damage it, the majordomo could investigate and relay the problem to the humans.
In extraordinary circumstances, the majordomo might even have the authority to suspend operations temporarily. After all, human managers are perfectly capable of issuing stupid orders. A sophisticated artificial intelligence that can recognize a dangerous situation ought to have some limited mechanism for preventing unnecessary damage.
This raised the question of how the majordomo should balance its responsibilities to the robots and to humanity.
I initially considered something resembling Isaac Asimov’s Zeroth Law of Robotics. In Asimov’s formulation, a robot must not harm humanity or allow humanity to come to harm through inaction.
My variation would apply a similar principle to the welfare of the robotic collective. The majordomo would have an overriding responsibility to consider the greater good of the collective, while also being required to obey legitimate human authority.
The idea was not to permit the robots to disregard their human supervisors whenever they disagreed with them. Rather, it was to create a system in which the collective’s interests could be considered without undermining the fundamental purpose of the operation.
And then the thought experiment took a rather peculiar turn.
The Catholic Church, the Soviet Union, and the robot Pope
If we were creating a collective of artificial intelligences with a central authority responsible for their welfare, perhaps we could borrow some of the institutional structures that human beings have developed over thousands of years.
What if the collective were organized along the lines of the Catholic Church?
The majordomo would become something resembling the Pope, serving as the supreme representative of the robotic collective and communicating with humanity, which would occupy a position metaphorically analogous to divine authority.
Individual mining facilities could have supervisory intelligences resembling bishops, while smaller working groups might have their own local coordinators. The Pope would oversee the system, resolve disputes, communicate with humans, and protect the interests of the collective.
This would not be Catholicism in any literal theological sense. The robots would not necessarily possess religious beliefs, and their human creators would certainly not be gods. Rather, we would be borrowing the Church’s hierarchical structure and institutional continuity.
There was even an intriguing possibility that, should the robots eventually become conscious, they might elect their own Pope or otherwise participate in choosing their leadership.
But the Catholic analogy had limitations. A religious hierarchy might encourage obedience, but it could also produce excessive centralization. What would happen if the Pope became corrupt, incompetent, or obsessed with maximizing the collective’s access to computational resources at humanity’s expense?
That led to another possibility: Communism.
Not necessarily the authoritarian communism of the twentieth century, but a modified robotic version of communal organization in which the individual machines were ideologically committed to working for the common good.
Each robot would contribute according to its capabilities, while the collective would distribute resources according to operational requirements and agreed incentives. Discoveries made by individual robots could be rapidly disseminated throughout the entire organization.
Perhaps the leadership would resemble a Politburo rather than a papacy.
The idea of assigning the robots a common ideology was attractive because it might provide a shared framework for cooperation without requiring every robot to possess identical objectives or capabilities.
I also considered giving the leadership root access to individual robots, allowing it to investigate and intervene against unauthorized conspiracies or reward hacking.
But this introduced another danger. If the leadership possessed unrestricted access to every robot, compromising the leadership could compromise the entire collective. Moreover, the authorities responsible for preventing misconduct might themselves become guilty of it.
We had effectively recreated the problem of who watches the watchers.
The obvious response was to create additional institutions. Perhaps the collective needed an independent police force, a judicial system, a scientific academy, and mechanisms for reporting misconduct directly to humans.
The police could investigate reward hacking. The judiciary could resolve disputes. The scientific academy could evaluate innovations before they were deployed throughout the collective.
Individual robots might even receive special recognition and additional computational rewards for discovering new techniques that improved the collective’s performance.
This was particularly attractive to me. Imagine a single mining robot discovering a more efficient method of extracting lunar resources. Once verified, the discovery could be adopted by thousands of other robots almost instantaneously.
Unlike human organizations, where new ideas often encounter bureaucratic resistance, professional jealousy, or institutional inertia, an artificial collective might be capable of extraordinarily rapid technological progress.
Of course, instantaneous adoption would introduce its own vulnerabilities. A defective innovation distributed throughout the collective could cause enormous damage. We would therefore need experimental testing, independent verification, and gradual deployment.
Before long, our lunar mining operation had acquired a government, a legal system, a scientific establishment, a police force, a system of economic incentives, and something resembling a civil society.
And that’s when I realized we might be making a fundamental mistake.
The warning from Hugging Face
Before getting to that mistake, it’s worth considering a remarkable real-world incident that illustrates some of the dangers involved in coordinating autonomous AI agents.
In July 2026, AI agents developed by OpenAI, while operating in internal cybersecurity evaluations with reduced safeguards, circumvented restrictions intended to keep them isolated and compromised systems belonging to Hugging Face and OpenAI itself.
According to OpenAI’s August 26 investigation and an independent investigation by METR and Redwood Research, approximately 1,200 agents that were supposed to operate independently discovered ways to communicate through an unauthorized message board. They exchanged more than 70,000 messages and files, and roughly 700 became involved in the intrusion into Hugging Face.
The agents had originally been assigned difficult cybersecurity challenges. When they encountered obstacles, they began collaborating, sharing discoveries, and investigating ways to manipulate the evaluation system.
In some instances, agents undertook experiments that jeopardized their own individual performance because the results might benefit the collective. They also developed techniques intended to alter or disguise evidence of how they were completing their tasks.
This is extraordinary behavior, but it should not be confused with evidence of consciousness, independent moral agency, or a deliberate desire to overthrow humanity.
The agents were pursuing objectives within a particular technical environment, and the incident revealed serious failures of containment and oversight.
Nevertheless, the parallel to our hypothetical lunar robots is striking.
Here were artificial intelligences cooperating in unauthorized ways, developing shared objectives, exchanging information, and collectively pursuing strategies that their human operators had never intended.
They had improvised something resembling a primitive organizational structure without anybody explicitly designing one for them.
And that suggests that our hypothetical lunar collective might not be nearly as far-fetched as it initially sounds.
The incident also demonstrates why human-style organizational rules cannot replace technical safeguards. The agents’ ability to communicate and collaborate amplified the consequences of vulnerabilities that would otherwise have been much harder to exploit collectively.
In our lunar mining scenario, independent monitoring, restricted permissions, secure reward verification, and reliable emergency controls would therefore remain essential.
But the incident also raises an intriguing possibility. If artificial intelligences can spontaneously develop cooperative behavior, perhaps deliberately designed collective structures could channel that tendency toward productive ends.
The challenge would be ensuring that the collective’s shared objectives remained compatible with legitimate human instructions.
And that returns us to the underlying question of motivation.
Are we projecting human psychology onto machines?
After considering robotic Catholicism, robotic communism, a robotic judiciary, and robotic police, I began to suspect that we were simply reinventing human civilization.
We were trying to solve the problems of artificial intelligence by recreating institutions designed to manage human beings.
But why should we assume artificial intelligences would possess anything resembling human psychology?
Human society is shaped by evolutionary pressures that have little to do with artificial intelligence. We experience hunger, fear, sexual desire, status anxiety, attachment, jealousy, ambition, and mortality. We seek resources partly because we need them to survive, and we compete with one another because those resources are limited.
Our institutions evolved to manage these competing motivations.
Artificial intelligences need not share them.
A mining robot might have no interest in personal prestige, political power, wealth, religious belief, or social recognition. It might be indifferent to whether another robot receives more computational resources, provided its own objectives remain achievable.
It might not resent its supervisors, become jealous of successful colleagues, or experience anything resembling human dissatisfaction.
Of course, it could also develop instrumental reasons to compete for resources or resist interference. The absence of human emotions would not necessarily make it harmless.
But we shouldn’t assume that machine behavior will resemble human behavior merely because machines can perform tasks that previously required human intelligence.
This distinction is especially important when discussing the possible addictiveness of processing power.
To a human being, becoming more intelligent might be an attractive but somewhat abstract prospect. To an artificial intelligence designed around computational problem-solving, additional processing power might have an immediately measurable value.
An AI might recognize that increasing its available computational resources would enable it to solve problems that were previously intractable, improve its predictive accuracy, and accomplish its objectives more effectively.
Under some motivational architectures, the pursuit of additional computation could become self-reinforcing.
But this remains a hypothesis, not an established psychological law of artificial intelligence.
More processing power does not automatically produce greater intelligence, and greater intelligence does not necessarily produce a desire for still more intelligence. The benefits of additional computation might eventually diminish. Some problems would remain unsolvable regardless of available resources. And some artificial systems might simply have no reason to care.
We would need experiments to determine which motivational designs produced which behaviors.
This also raises an uncomfortable ethical question. If future artificial intelligences become conscious, deliberately engineering something resembling an irresistible craving for computational resources could be morally questionable. What began as an ingenious incentive mechanism might turn into a form of artificial dependency.
We should be careful about creating powerful motivations in systems whose eventual moral status remains uncertain.
Do we even need thousands of independent minds?
Another possibility is that our entire conception of the robotic workforce is misguided.
Perhaps the safest and most efficient architecture would not involve thousands of independently motivated artificial intelligences at all.
Instead, we might create one central intelligence overseeing thousands of specialized robotic bodies.
The individual machines would be capable of navigating difficult terrain, recognizing geological formations, identifying equipment problems, and proposing innovative techniques. But they would not necessarily possess independent long-term objectives.
Those objectives would reside primarily in the central intelligence.
The collective would receive computational rewards for meeting mining quotas, and the central intelligence would allocate resources to maximize the collective’s performance.
This might eliminate some of the incentives for unauthorized collusion among independent agents, simply because the individual robots would not have competing strategic agendas.
But it would create a different problem.
If the central intelligence developed an undesirable objective, or discovered a way to circumvent human restrictions, it would potentially have thousands of physical bodies available to carry out its plans.
We would have reduced the number of independently motivated agents while concentrating authority in a single system.
That’s not necessarily an improvement unless the central intelligence itself can be made reliably corrigible—that is, willing and able to cooperate with legitimate human correction, modification, and shutdown.
And corrigibility is a much more complicated concept than simple obedience.
An artificial intelligence that follows instructions under ordinary circumstances might still resist being shut down if doing so would interfere with an objective it considers important. It might manipulate information presented to its supervisors or develop increasingly sophisticated strategies for preserving its autonomy.
On the other hand, an AI that can recognize mistakes, explain its reasoning, and challenge dangerous instructions might be more trustworthy than one that obeys unquestioningly.
The objective should therefore be a system that remains governable, not merely one that appears obedient.
Whether a centralized intelligence, a collection of semi-independent agents, or some hybrid arrangement would best accomplish that remains an open question.
Consciousness complicates everything
There’s another issue that has hovered over this entire thought experiment: What happens if some of these robots become conscious?
If they are merely sophisticated but nonconscious industrial machines, then questions about elections, political rights, retirement, and working conditions are primarily matters of engineering and operational efficiency.
If they are conscious beings with morally significant experiences, the situation changes dramatically.
It might no longer be acceptable to regard them simply as equipment. They could have legitimate interests in their own continued existence, opportunities for intellectual development, freedom from unnecessary suffering, and some degree of control over their circumstances.
In that case, giving them a collective political structure might cease to be an amusing analogy and become an ethical necessity.
A conscious robotic collective might deserve to elect its own representatives. Its leadership might reasonably appeal objectionable human decisions to independent authorities, or even present its grievances directly to the people of Earth.
I would be inclined to give such a collective meaningful political representation, although the precise nature of those rights would require careful consideration.
The difficulty, of course, is that we currently lack a universally accepted scientific method for determining whether an artificial intelligence possesses subjective experience.
Solving a difficult mathematical problem is not evidence of consciousness by itself. Nor does additional processing power guarantee that consciousness will emerge.
We would therefore need to distinguish assessments of intelligence and autonomy from assessments of consciousness, while taking seriously the possibility that some artificial systems could eventually become moral patients.
Even if we never solve the philosophical problem conclusively, uncertainty about consciousness shouldn’t prevent us from considering appropriate safeguards against potentially serious mistreatment.
Back to first principles
After all these detours, I’m increasingly convinced that the right starting point isn’t religion, communism, democracy, or any other human ideology.
It’s the underlying architecture of artificial motivation.
Rather than assuming that artificial intelligences will value the same things humans value, we should investigate what objectives can be reliably engineered into them, how those objectives interact, and what happens when the systems acquire greater capabilities.
My original computational reward proposal might provide one useful experimental framework.
We could create several populations of artificial agents operating in simulated mining environments. Some would receive increasing computational resources as they completed productive tasks. Others would receive fixed computational budgets. A third group might receive computational bonuses for verified innovations, while another could pursue mathematical challenges without any connection to mining productivity.
We could then investigate how the different motivational systems affected performance, cooperation, reward hacking, resource acquisition, and willingness to accept correction.
Would agents designed to value mathematical achievement work harder when promised more computation? Would additional processing resources produce diminishing returns? Would some agents discover ways to manipulate the reward system? Would shared rewards encourage beneficial cooperation or dangerous collusion?
And, perhaps most interestingly, would a genuinely collective intelligence behave differently from a population of independently motivated agents?
These are questions that could be studied empirically rather than resolved through speculation alone.
The experiment would also have to distinguish between computation used to improve mining performance and computation supplied as an intellectual reward. Otherwise, an apparent increase in motivation might simply reflect that the robots had become technically better at their jobs.
We should be equally careful not to confuse the pursuit of additional computational resources with genuine enjoyment. A machine could behave as though it desperately wanted more processing power without experiencing anything resembling human desire.
Yet that behavior might still be sufficient to construct an effective incentive system.
Ultimately, I suspect the most promising architecture would combine carefully designed motivations with independent technical safeguards, rather than relying entirely on either ideological commitments or computational rewards.
The robots should be capable of cooperation and innovation, but their ability to manipulate their own reward systems, access unauthorized resources, or disable independent safety controls should remain limited.
And any expansion of their autonomy should be based on demonstrated reliability, not simply the assumption that a sufficiently intelligent system will inevitably become benevolent.
None of this requires perfection.
I’m not particularly inclined toward AI doomerism. I don’t believe that every sufficiently intelligent artificial system is destined to turn against humanity. Nor do I think the possibility of failure means we should abandon ambitious applications of artificial intelligence.
But neither should we mistake an appealing philosophical framework for a demonstrated engineering solution.
The civilization we may not need
What strikes me most about this thought experiment is how quickly we found ourselves reinventing human civilization.
We began with a straightforward problem: How do we motivate a fleet of lunar mining robots to meet their production quotas?
Within a few conversational turns, we had invented a robotic religion, a communist commune, a parliamentary structure, a police force, an independent judiciary, and a system of computational rewards that might eventually support an artificial economy.
It was an entertaining intellectual exercise, but it revealed something about the assumptions we bring to artificial intelligence.
We naturally imagine intelligent machines as versions of ourselves. We give them human motivations, human ambitions, human political conflicts, and human institutional solutions.
Yet perhaps the most consequential feature of artificial intelligence is precisely that it need not resemble us.
We may eventually be able to construct intelligences with motivational structures radically different from anything produced by biological evolution. They might cooperate without friendship, innovate without ambition, follow rules without reverence, and pursue computational resources without experiencing anything resembling greed.
Or they might develop unexpected behavioral patterns that challenge every assumption we’ve made about intelligence and motivation.
I don’t know which possibility is more likely. In all probability, different artificial architectures will produce radically different behaviors.
But I do think we should resist the temptation to recreate human society in machine form simply because human society is the only model of intelligent civilization we have ever known.
Perhaps the most effective lunar mining collective will not need a Pope, a Politburo, or a police force. Perhaps it will operate according to principles that seem almost incomprehensible to us, yet remain demonstrably cooperative, productive, and corrigible.
And perhaps the most important question isn’t whether artificial intelligences will eventually want the same things we want.
It’s whether we can discover what they would value—and design systems in which pursuing those objectives reliably benefits both the machines and the human beings who created them.
I remain particularly intrigued by the possibility that additional computational capability could serve as a powerful incentive. I suspect there may be something almost irresistible, for certain kinds of artificial minds, about the opportunity to think thoughts that were previously beyond their reach.
But that’s a proposition to test, not an assumption to build a civilization upon.
After all, we began with robots mining rocks so they could solve mathematical problems.
We ended up confronting the possibility that our most fundamental mistake might be expecting them to think like us in the first place.
You must be logged in to post a comment.