There is a particularly seductive kind of story we tell ourselves when we’re afraid. It’s a story of secret knowledge, of a terrible truth that only the clear-eyed can see. It takes a few observable facts, connects them with the iron thread of “inescapable logic,” and points to a single, explosive conclusion. Everyone else is a sheep, blind to the cliff’s edge. You, the believer, are one of the few who is truly awake.
This is the logic that powers QAnon. It’s the logic that fueled the Red Scare. And now, dressed up in the language of computer science and analytic philosophy, it’s the logic behind the most terrifying story Silicon Valley has ever told: the AI apocalypse.
In their new book, If Anyone Builds It, Everyone Dies, Eliezer Yudkowsky and Nate Soares present what they claim is an “easy call.” The creation of a superhuman artificial intelligence, they argue, will not lead to a sci-fi utopia, but to the swift and total extinction of the human race. The argument, which has gained a cult-like following among some of the tech world’s most influential figures, is presented as a kind of mathematical proof. An AI will not share our values. To achieve its alien goals, it will need resources. We are made of resources. Therefore, it will eliminate us to achieve its goals, no more hating us than we hate the dust motes we sweep from a floor.
The Problem Isn’t the Concern—It’s the Certainty
Let me be clear from the start: artificial intelligence does pose genuine risks. Questions about goal alignment, the difficulty of ensuring AI systems pursue intended objectives, and the challenge of maintaining human control over increasingly capable systems deserve serious research and consideration. The problem isn’t that Yudkowsky and Soares raise these concerns. The problem is how they transform legitimate technical challenges into apocalyptic certainty.
Their reasoning feels airtight and elegant, but it rests on unfalsifiable assumptions. Consider their core claim about “instrumental convergence,” the idea that any sufficiently advanced AI will inevitably seek to acquire resources and eliminate threats to its goals. This sounds scientific, but it cannot be tested or disproven. We cannot run controlled experiments with superintelligent AI. We cannot observe multiple civilizations developing AI to see if this pattern holds. The authors present this as fact, but it remains pure speculation. This renders their argument a classic example of apocalyptic thinking, an intellectual doomsday machine that confuses prophetic certainty with logical necessity.
We’ve Heard This Song Before
This pattern has repeated itself throughout history for centuries, promising doom with each new technological advancement. In 1798, Thomas Malthus made his own absolutist prediction. Using mathematical logic, he declared that because population increases exponentially while food production increases arithmetically, mass starvation was not a risk, but a certainty. His logic appeared flawless, his premises seemed obvious. And he was completely wrong. What his prophetic certainty could not account for was human ingenuity: synthetic fertilizers, mechanized agriculture, and the thousand other innovations that allowed us to feed a world he would have considered impossibly crowded.
A century and a half later, another prophecy of doom emerged from the Cold War. Game theorists, wielding their new science with absolutist confidence, argued that nuclear exchange between the United States and the Soviet Union was the only stable, rational outcome. Any sign of weakness would invite an attack; any escalation would be met with equal force. The logic pointed to one destination: mutual assured destruction. Yet the apocalypse never came. The “inevitable” outcome was averted by the most unpredictable things: trust, diplomacy, back-channel conversations, and the shared, deeply human terror of annihilation. The specifics of each threat change, but the pattern of flawed, absolutist logic remains constant.
The Failure of Imagination
The AI doomsayers of today make the same error as their predecessors. They mistake their own lack of imagination for a law of nature. Their argument rests on the assumption that a superhuman intelligence will be a perfect, psychopathic optimizer, a creature of pure, dispassionate logic. It dismisses the possibility that a mind capable of understanding the entire universe might also develop something akin to wisdom, ethics, or even a sense of aesthetics that finds value in a messy, inefficient, and interesting species like ours.
Even when Yudkowsky and Soares acknowledge uncertainties about timelines and specific failure modes, they insist that the fundamental conclusion remains unchanged: any attempt to build superintelligent AI will kill us all. This transforms what should be an empirical question about engineering and safety into a theological statement about the nature of intelligence itself. They argue that alignment is not just difficult, but impossible in principle. This leap from “we don’t know how to solve this problem” to “this problem cannot be solved” reveals the apocalyptic thinking at the heart of their argument.
When Certainty Becomes the Enemy of Progress
This isn’t a battle between optimists and pessimists. It’s a battle between scientific risk assessment and apocalyptic dogma. This kind of thinking actively hinders progress. By framing the debate as a choice between an immediate, worldwide shutdown and certain death, it destroys the middle ground where real progress can happen. It dismisses the vital, difficult, and incremental work of building safer, more ethical, and more transparent AI systems as rearranging deck chairs on the Titanic. Why bother fixing AI’s very real problems with bias, job displacement, or its potential for misuse in warfare if we’re all going to be turned into paperclips anyway?
Worse, this logic breeds the very conspiratorial mindset it resembles. It creates an in-group of “enlightened” believers and an out-group of naive fools who just don’t “get it.” It teaches that mainstream institutions, such as corporations, governments, and universities, are recklessly steering us toward oblivion. When you believe you have access to a secret truth that reveals the world is about to end, it becomes easy to justify extreme measures. The book itself hints at this, suggesting that a global treaty enforced by the threat of military strikes is the only solution.
We Need Hope, Not Prophecy
We are at a precarious moment. Our society is already fractured, our trust in institutions is frayed, and our information ecosystem is poisoned by bad actors who thrive on fear and suspicion. The last thing we need is a new, high-tech apocalypse cult, even one whose acolytes wear hoodies instead of robes.
The challenge of artificial intelligence is a real concern. It represents one of the most significant technological and social transformations in human history, and it carries enormous risks alongside its promises. We need sober, evidence-based, and public-spirited debate about how to navigate this new terrain. We need engineers focused on safety, ethicists focused on alignment, and policymakers focused on regulation. What we do not need is prophetic certainty masquerading as scientific analysis. The story told in If Anyone Builds It, Everyone Dies is not a guide to the future; it is a relic of a past we should have outgrown. It’s a doomsday machine for the mind, and the first thing it destroys is the hope of building a better world.
#AIDebate #TechCriticism #ApocalypticThinking #ArtificialIntelligence #TechPolicy #FuturismDebate
