When Technology Becomes Religion
On artificial intelligence, Roko’s Basilisk, and belief in a future that does not yet exist
I was listening to The Daily from The New York Times when an expression came up that I could not quite catch.
I am hard of hearing. When a word does not reach me, I do what I assume all human beings do, only perhaps a little more often: I try to make sense of what I actually hear.
“Roco basselement,” I heard.
It had to be wrong. But the words had already become something. I had interpreted the sounds before I knew what they meant.
It turned out to be Roko’s Basilisk.
When I understood what it was, it struck me that this small mishearing almost contained the very problem I was listening to. I had encountered something unfamiliar with what I already had at my disposal. I had formed a preliminary understanding before I knew what the matter was about.
And now the conversation concerned people trying to understand something far more difficult:
An intelligence that does not yet exist.
A Future Judge
Roko’s Basilisk is a thought experiment that emerged in the LessWrong community in the early 2010s. The idea is roughly this: At some point in the future, human beings may create an artificial superintelligence with capacities we can barely imagine. This intelligence might have an interest in being created as early as possible. It could therefore conceivably punish people who knew that it might be created but nevertheless failed to contribute to bringing it into existence.
It is easy to dismiss this as science fiction.
But the basilisk does not need to exist in order to influence how people act.
A Future That Acts Backwards
We constantly act on the basis of ideas about the future. We save for old age. We educate ourselves for a profession we do not yet have. We plant a tree that may not provide shade until others are the ones sitting beneath it.
But Roko’s Basilisk reverses this relationship in a peculiar way. A possible future intelligence is imagined as though it were already watching us. Our actions today acquire meaning in light of how this intelligence might one day judge them.
Something that does not yet exist begins to act upon what does exist.
This is where the resemblance to religion appears.
For centuries, human beings have lived with the idea that their actions are seen from a perspective that transcends their own. Religions have developed languages of judgement, salvation, damnation, and hope beyond the life of the individual.
Roko’s Basilisk reproduces parts of this structure, but without God.
In God’s place stands the machine.
The Secular Judge
The comparison should not be taken too far. Religion is far more than fear of punishment. Religious traditions encompass community, ritual, love, hope, mercy, ethics, and conceptions of meaning that cannot be reduced to judgement.
Still, the structure is difficult to overlook.
In some visions, a future superintelligence acquires qualities that once belonged to the language of religion. It may know more than any human being, solve problems we cannot solve ourselves, and perhaps help combat disease and extend human life. In the darkest visions of the future, it may destroy us.
Salvation and damnation return as technological possibilities.
With the basilisk comes judgement as well.
There is a distant kinship here with Pascal’s famous wager. Pascal examined how a human being should relate to an uncertain possibility when the stakes were understood to be infinite.
Something similar happens when superintelligence is discussed. Even an uncertain future can acquire considerable power over the present if the consequences we imagine are great enough.
Belief Comes Before Its Object
We have no artificial superintelligence of the kind these stories describe. We do not know whether one will ever exist, much less what it would be like if it did.
Yet the stories already exist.
Some envision a technology capable of helping solve problems humanity has struggled with for generations. Others see a possible existential risk. Both sides also seek to ground their arguments in research, mathematics, and technological development. It would therefore be unreasonable to call the entire debate a religion.
But human beings do not live by facts alone.
We understand facts through our ideas of what they mean.
When Belief Becomes Capital
And now something decisive happens.
The idea has acquired money.
The largest technology companies are investing hundreds of billions of dollars each year in data centres, computer chips, networks, energy, and other infrastructure connected to artificial intelligence. Over several years, these investments move into the trillions of dollars.
The idea becomes material.
It becomes data centres, power lines, cooling systems, factories, computer chips, jobs, debt, and expectations of future returns.
And once the capital has been invested, inertia develops.
A company that has invested tens of billions of dollars cannot simply decide to wait a few years and see what happens. Investors expect returns. Employees have been recruited. Power has been purchased. Competitors continue. Governments regard the technology as economically and strategically important.
So who slows down first?
One company may fear that its competitor will continue. The United States may fear that China will continue. Europe may fear being left behind while both move ahead.
Everyone may want safer development while each has powerful reasons not to be the only one to slow down.
And here the paradox appears.
Several prominent AI researchers and technology leaders have warned that highly advanced artificial intelligence could, in the worst case, pose an existential risk to humanity. This does not mean that we know artificial intelligence will destroy humanity. We do not. But some people with extensive knowledge of the technology believe that the possibility is serious enough to be taken seriously.
Yet fear itself may also accelerate development.
If the others continue, do we dare to stop?
The same future can therefore become an argument both for slowing down and for accelerating.
And the money makes a difference.
An idea is no longer merely an idea when society begins to build its material world around it.
Religions built churches.
Artificial intelligence builds data centres.
A data centre is, of course, not a sacred space, and an investment is not a profession of faith. But both tell us something about what human beings are prepared to do materially on the basis of what they believe matters.
We build for the future we believe is coming.
And once we have built, that future becomes harder to choose against.
The Hermeneutic Problem
Here I return to my own mishearing.
“Roco basselement.”
That was not what had been said. But I could not hear an empty space. The sounds were assembled into something I already knew. Before I could correct my understanding, I first had to have an understanding that could be corrected.
Gadamer showed that understanding never begins from a neutral point. We encounter the new with an already existing horizon of understanding. Our prejudgements – in the hermeneutic sense – are not merely obstacles to understanding. They make understanding possible, but they must be capable of being put at risk and corrected in our encounter with the matter itself.
Perhaps this applies to artificial intelligence as well.
We do not encounter the superintelligence of the future.
We encounter our ideas of it.
And when we try to imagine an intelligence that surpasses our own, we reach for images we already know:
The Creator.
The Judge.
The Saviour.
The All-Knowing.
Old ideas acquire new technological forms.
Kierkegaard and Uncertainty
In Fear and Trembling, Kierkegaard, through Johannes de silentio, places Abraham in relation to something that cannot be made certain through universal reason. The narrator tries again and again to understand Abraham – and discovers the limits of his own understanding.
The point here is not to turn artificial intelligence into a question of religious faith. That would push the comparison too far.
What matters is the situation in which human beings find themselves when action is required before certain knowledge is possible.
We must make decisions about research, investment, regulation, and risk without being able to know what the technology we are discussing will one day become.
The uncertainty cannot be removed.
The question is what we do with it.
Who Creates Whom?
Here the circle closes.
We imagine the future. The idea generates investment. Investment builds infrastructure. Infrastructure makes further development possible while simultaneously creating economic interests in its continuation. That development, in turn, makes the idea of an approaching superintelligence appear more plausible.
The future acts back upon the present.
The present then begins to build the future it imagined.
The hermeneutic circle has acquired computer chips, concrete, electricity, and capital.
Human beings create the machine.
But the idea of the machine is already helping to shape the human world.
The Basilisk Already Exists
Roko’s Basilisk does not exist as the future superintelligence described by the thought experiment.
But the name exists. The story exists. The fear exists. The hope exists.
And the capital exists.
And so does the effect.
I began by hearing a word that had not been spoken. “Roco basselement.” Only when I discovered the mistake could I begin to examine what I had actually heard.
Perhaps we need the same movement when we encounter stories about artificial superintelligence: to examine our ideas before we turn them into truths – and before our investments turn them into a future that becomes increasingly difficult to choose against.
For long before an artificial superintelligence might learn to interpret the human being, the human being has begun to interpret it.
And between what we think we hear and what is actually being said, there is a small space.
Understanding begins there.
This essay was written in a conversation with Claude/Anthropic and OpenAI/ChatGPT