Nine days after Jacob Coxon made a spectacle of quitting Anthropic, warning that many of the people building advanced AI sincerely believe it could kill us all within the decade, the New York Times responded with an editorial that had an oddly, almost disconcertingly cheerful title: “Humanity Has Avoided Apocalypse Before. Let’s Do It Again.”
Drawing an analogy with postwar efforts to control nuclear weapons, the editorial argued that “Human society can restrain a dangerous innovation once it decides to”. But this optimism, while a welcome antidote to the current AI fatalism, looks less assured on closer inspection.
For “once it decides to” is doing rather a lot of work. The hoped-for sequence sounds perfectly rational: society recognises the dangers posed by some intoxicating new technology, decides they require restraint, and then gets on with the difficult business of coordination and regulation. Missing from this story is any account of what has to happen before a society can recognise that a powerful and promising innovation contains threats sufficiently real—and sufficiently morally urgent—to justify curbing it.
The same postwar history the Times invokes as grounds for optimism also suggests, more darkly, how fraught that process of recognition can be. The philosopher Günther Anders, writing after Hiroshima, was interested in just this problem. Modern technology, he argued, had opened a gap between what human beings were capable of producing and what they were capable of imaginatively comprehending. Our powers had become enormous while our moral imaginations remained stubbornly small. We could know, propositionally, that a series of events might destroy millions or even everyone alive, and yet still fail to make that possibility morally real to ourselves. Anders called this disparity the “Promethean gap”.
The difficulty seems paradoxical. Surely the greater the catastrophe, the easier it should be to fear it and work to avoid it? Yet historical experience shows us it is often the other way round. One dead child is imaginable. A thousand dead children become a statistic. The destruction of a city can be photographed, while the extinction of humanity cannot. In this way, it becomes possible to talk—and write—about the end of humanity while barely imagining anything at all.
The existential worry around rogue AI seems a case in point. While nuclear war has its mushroom clouds and pandemics have their hospitals and morgues, “loss of control over superintelligence” is much more abstract. It arrives in the language of AI model behaviour, probabilities, alignment failures and chains of cascading events that nobody has experienced before. The threats may be grave and global, but our imaginative grasp of them remains thin.
The other side of the ledger, meanwhile, appears all too richly imagined by those accelerating AI. Diseases are cured, work disappears, abundance arrives, humanity escapes Earth and perhaps even biology itself. Sam Altman has spoken of uploading his brain to the cloud; Elon Musk wants humanity to become interplanetary; Silicon Valley venture capitalist Marc Andreessen has written of becoming “technological supermen”.
The problem, then, may not be a lack of imagination, but a clash between two quite incompatible ways of imagining the end. One understands catastrophe as a possible passage to something greater. The present order is limited, decaying, frustratingly human; beyond it lies superintelligence, posthumanity, cosmic expansion, perhaps even immortality. Here, the greatest danger may not be moving too fast but failing to move fast enough. In Andreessen’s Techno-Optimist Manifesto, this logic reaches its endpoint: hesitation itself becomes a source of preventable suffering and death. “We believe any deceleration of AI will cost lives.”
The other moral imagination was forged after the catastrophes of the 20th century. Its governing suspicion is almost the reverse: beware visions of the future magnificent enough to make present people look expendable. Hannah Arendt’s language of human “superfluousness” has purchase here. She was writing about totalitarianism, of course, not artificial intelligence, but the question she poses still applies. What happens when some supposedly inexorable movement of history comes to matter more than the particular human beings living through it?
The postwar answer was not that humanity must never take bold risks or attempt radical change. It was that the allure of an imagined future cannot, by itself, justify treating particular human beings as expendable in its pursuit. That principle becomes harder to hold on to when the promised destination is almost unimaginably grand.
Of course, there are more prosaic reasons why AI development continues despite frightening warnings. Money matters, as do geopolitical competition, sunk investment and the familiar argument that, if responsible companies stop, irresponsible ones will simply carry on. But such pressures do not fully explain why pressing ahead can appear the morally courageous choice to many of those in charge.
The 20th century gave us “Never again”, an ethics built partly from memory. It says: look, this happened: people suffered, survivors testified; here are the photographs, the artefacts; do not allow it to happen twice. The risks posed by AI ask something different of us: to recognise catastrophe before memory, testimony and ruins have made its meaning vivid for us—without waiting for disaster itself to supply the lesson.
