One of the many odd things about the recent frenzied debate and political scrambling over AI’s alleged existential threat to humankind is that it was sparked by the resignation of a junior researcher. On X, 27-year-old Jacob Coxon said that he was leaving the AI company Anthropic, creator of the chatbot Claude, because it and his previous employer OpenAI (home of ChatGPT) were not acting responsibly. “They are racing straight to self-improving superintelligence and gambling with our lives,” Coxon posted.
“The people building AI earnestly believe that it could kill us all by the end of the decade,” he continued. “No other human activity poses this level of danger.” His comments were supported by another Anthropic researcher, Evan Hubinger, who estimated the probability of humanity being wiped out by AI in the next decade as greater than 10 per cent.
Another strange aspect of the current panic is that the claims made by Coxon and Hubinger are nothing new. Warnings of humanity’s demise at the hands of AI have been voiced for years, notably in the 2014 book Superintelligence by the philosopher Nick Bostrom. Stephen Hawking’s regular prophecies were treated with guru-like gravitas. The dangers have already been discussed in high-profile summits. Last year Nate Soares and Eliezer Yudkowsky, respectively the president and founder of the Machine Intelligence Research Institute, published a book titled If Anyone Builds It, Everyone Dies, which summarises their terrified (and some say terrifying) view of AI superintelligence. So why did Coxon’s posts make such an impact?
Perhaps they were a final straw amid widespread growing unease about the rate of advance in AI and the effects it is having on our lives. Yet in all the column inches devoted to Coxon’s and Hubinger’s comments, there has been remarkably—I would say dismayingly—little interrogation of their provenance. The prevailing assumption is that since these are the people making the technology, surely they know best.
Coxon and Hubinger might be mere foot soldiers in the effort to make super-powerful AI, but one can’t say that of Geoffrey Hinton, the British computer scientist who won a Nobel prize for his work in developing AI. Hinton resigned from Google in 2023 so that he was freer to “talk about the dangers of AI”, and he too believes it poses a significant risk of human extinction. When Hinton appeared on Newsnight on 9th September he told Victoria Derbyshire that Hubinger’s estimate of greater than 10 per cent was “not unreasonable”. She could respond only with a stunned “Wow. Oh my god.”
But we have been groomed for years to respond in this way. The AI industry has been permitted to set the terms of debate, capitalising on a long cultural history of narratives about “intelligent machines”. At the same time, political and cultural discourse has few resources for critiquing the claims of scientists and technologists. The tech giants of Silicon Valley have exploited these vulnerabilities ruthlessly, cultivating the myth of super-smart men (it’s always men) who create technologies beyond the comprehension of most of us. So successful have they been that they now seem to believe this myth themselves. It is precisely because AI is the product of their special genius, they imply, that it is so dangerous. Couple this to a capitalistic story that makes technical advance tantamount to an inevitable law of nature, and the narrative capture is complete.
The real counter-argument to their claims is not about the numbers or probabilities, but the framing. Linguist Emily Bender of the University of Washington has lamented “how expertise is being constructed in this news cycle” about AI. Those in the industry, whether boosters or doomers (and the distinction is less clear than you might suppose, as both ultimately proclaim the technology’s awesome power), are met with wide-eyed credulity (“Wow!”), while critical voices are either ignored or relegated to the “sceptics box”. “We need many different kinds of expertise to counter the power being amassed by tech companies,” Bender says.
‘If we build AI regulations around a future fantasy, we lose sight of where the real power lies’
Before deconstructing the discourse, let’s be clear: the risks posed by AI are very real, and indeed already evident. They range from pollution of the infosphere with slop to the theft of intellectual property, from energy-hungry data centres and the hidden human costs of data farming, to the displacement of jobs. One of the biggest concerns is simply the hegemonic power that a society-wide infatuation with AI gives to an industry that seems ever less concerned about privacy, copyright, social welfare or democracy. And in the hands of malign actors, AI has the potential to wreak havoc—disrupting essential internet resources, spewing out disinformation, even (though this could be hard to implement in practice) assisting in the construction of bioweapons.
None of these risks rely on AI becoming so powerful that it displaces or eliminates all humans. Plenty of informed people have been warning about these dangers, and calling for oversight and regulation, for years. To many of them, the “existential risk” narrative that so excites the media is a distraction. “If we build regulations around a future fantasy,” writes Jack Stilgoe, an expert on science and policy at University College London, “we lose sight of where the real power lies and give up on the hard work of governing the technology in front of us.”
Machines like us
The threat to humanity from artificial beings and machines is an old tale, with roots in Mary Shelley’s Frankenstein (1818) and Samuel Butler’s “The Book of the Machines” in his utopian novel Erewhon (1872). But it was the 1920 play RUR by Czech writer Karel Čapek that brought these stories into a form appropriate for the age of industrial mass production. Čapek used the Czech word for indentured labour—robota—for the beings produced on an assembly line by the eponymous company Rossum’s Universal Robots, which rebel and wipe out humans. It’s a short leap from there to the rogue replicants of Blade Runner and the malevolent Skynet of the Terminator franchise.
The central question raised by these tales is to what extent humans are, if at all, distinct from our artificial creations. Shelley and Čapek framed that issue in terms of a soul; today it’s a matter of whether AI has consciousness or awareness. Hinton’s suspicion that today’s AI does so is shared by very few cognitive scientists.
One of those scientists, Anil Seth of the University of Sussex, thinks we may be confused by the fact that chatbots manipulate words rather than, say, numbers. “When we identify conscious experience with seemingly human qualities like intelligence and language, we become more likely to see consciousness where it doesn’t exist,” he says.
Ever since its inception, the field of AI has held a view of the human mind strikingly at odds with those who study it. The term “artificial intelligence” was coined by mathematician John McCarthy, largely for want of a better phrase, for a conference convened in 1956 at Dartmouth, New Hampshire, to discuss the possibility that the newly invented digital computers might be capable of displaying something like thought.
It tells us a lot about the participants in the early field that they considered an ability to play chess a good hallmark of human-like intelligence. Some of those researchers anticipated there would be machines with human-like powers of thought within a couple of decades. In the event, it took until 1997 for IBM’s Deep Blue to defeat the world chess champion Garry Kasparov—and Deep Blue was purpose-built for the job and its victory a matter of number-crunching. Any suggestion that the machine was conscious would have been ridiculed.
Still, the conflation of human and machine persists. In this view, if AI has already attained human-like capabilities such as awareness, this is because we are merely fancier versions of the same sort of machine. You say that AI is just a glorified auto-complete? Well, says computer scientist Stuart Russell, a prominent voice in AI safety (his 2021 Reith Lectures were on the subject), “humans when speaking are just next-word predictors too!” That will be news to linguists and cognitive scientists.
As AI ethicist Shannon Vallor, who previously worked at Google, says in her book The AI Mirror, AI “threatens us from within our humanity” because of these invitations to reimagine our humanity in the machine model. “What do you do when you are a faulty machine?” Vallor asks. “Look for a better one.” Such false equivalence bolsters the notion that AI can do what we do, but better (and with no trade unions).
The myths of AI
One of the industry’s major gambits is artificial general intelligence (AGI), which is often presented as its holy grail. AGI has typically been defined as an ability to perform any cognitive task at least as well as the average human. But ask proponents of AGI about whether it would display human qualities such as empathy, or emotional intelligence, and the response tends to be: no, not those things, obviously!
Rather, AGI merely encompasses tasks that can be measured and quantitatively graded—or, as OpenAI CEO Sam Altman blithely proclaimed, any task for which the machine can replace a “median” human. In other words, AGI refers to a grab-bag of those human cognitive abilities that are perhaps technically achievable by AGI, and commercially useful, such as the ability to do maths or having spatial awareness, but which, as a description of human general intelligence, are utterly arbitrary and scientifically meaningless.
In contrast, the real successes of AI in recent years come from models that do one thing very well, such as Google DeepMind’s AlphaFold2, which won a Nobel prize for the company’s cofounder Demis Hassabis. AlphaFold2 can predict the three-dimensional structure of a protein molecule—an important determinant of its biochemical function—from the sequence of the string of amino acids that constitute it.
AI’s ability to accelerate scientific research and discovery lies in highly specialised models—predicting, say, the effect of modifying an organism’s genes, or helping to find materials with new properties. Such models pose no obvious safety risks, let alone existential threats. AlphaFold2 has no incentive to take over the internet and no capacity to try. Slowing and regulating the development of AI need not inhibit its potential to help science.
Talk of ‘superintelligence’ allows tech CEOs to emphasise their commitment to AI safety while reminding us how brilliant they are
But even here that potential is routinely hyped to a gullible media. Who is going to question Hassabis when he says that, thanks to models like AlphaFold2, we might cure all disease within a decade? Perhaps reporters should also ask for the views of those who work on mitigating disease and developing drugs, to whom claims like this are exasperating nonsense. Likewise, AI will not solve climate change by disclosing new green technologies, or jump the hurdles to nuclear fusion, because the real obstacles—such as securing political, consumer or market support for a new technology—are not of the sort that AI can usefully tackle via computation.
The AGI narrative has become entangled with speculation about its ability to reason like us, to develop underlying models of what the world is actually like—of the kind with which we intuitively navigate our lives—and even to show signs of consciousness. There is no harm in asking such questions, and how AI “thinks” is an important matter. But we would be foolish to suppose that the AI industry is best placed to evaluate it. And there are good reasons to suppose that expertise in the technical aspects of AI, far from inoculating you against credulity, has the opposite effect.
Studies have shown that those within the industry were more liable to say that AI models pass the famous Turing Test (by giving answers that are indistinguishable from those of humans) than those outside it. Further, Alan Turing’s 1950 thought experiment, while a beloved part of AI lore, is considered of questionable value by many with expertise in cognition and philosophy of mind.
Talk of AGI has been strangely muted in the current controversy; now, the buzzword is “superintelligence”. But this term is no better. Bostrom’s book of that title defined it in much the same way as AGI, although these days it seems to rest less on claims of generality; superintelligence is merely AI that is way smarter than you somehow. (By some measures that would be true of the mainframe computer systems of the 1970s—it’s precisely why we made them.)
The point of that word is aggrandisement. It enables tech CEOs to talk earnestly about their commitment to AI safety while at the same time reminding us how brilliant they are. As Bender has warned: “Resist the urge to be impressed.” The conceit of the superintelligence discourse is that of a Nietzschean machine-being: an all-powerful supremacist who considers “median” humans irrelevant distractions, unworthy of existence. Why tech leaders think a hypothetical superintelligence should have such a negative attitude towards most of us is not hard to deduce: it’s a classic case of projection.
Bostrom’s book introduced a now famous thought experiment showing how a superintelligent AI with a narrow objective could wreak existential harm. Suppose, he said, we assign it the task of making as many paperclips as it can. Given sufficient resources, the machine might end up pulling apart the entire planet and all its inhabitants to rearrange their atoms into paperclips. How could we avoid such a misalignment of the superintelligence with our own interests?
Bostrom’s paperclip apocalypse was, however, imposed by fiat. We are asked to imagine a device so powerful that it can command any resources, outwit any attempts to stop it, is somehow able to manipulate matter at the atomic scale, and yet too stupid to understand any command not to harm humans. (Imagining human controllers dumb enough to make such a machine and give it such powers is arguably less of a challenge.)
This much vaunted “alignment problem” for AI is a fancy and self-serving reframing of the familiar notion that a new technology can have unintended consequences (witness the motor car and, say, the rise of out-of-town retail parks). While “alignment” implies a convergence between two autonomous agents, what we should really ask is whether AI’s builders have adequate control over it. The alignment framing directs us away from that question, from corporate and individual responsibilities, and instead presents the companies as nobly trying to persuade the obstinate AI to renounce unsafe behaviour.
Secret agents
Consider how this kind of framing shaped the response to Coxon’s statement. Earlier this year, researchers at OpenAI created an AI model with “agential” characteristics, which could to some extent determine its own strategy in pursuing an assigned goal. When tested for cybersecurity risks in a supposedly isolated “sandbox” environment, the model found a way to access the internet and to hack into the data platform Hugging Face. (It has since emerged an OpenAI model enabled an even more egregious hack when in June it accessed the Australian government’s universal healthcare system, Medicare.)
The complexity and apparent ingenuity of the AI agents in subverting the designers’ intentions and instructions was genuinely alarming, and it is tempting to impute an almost nefarious, deceitful determination in the model’s actions. But most discussions of the incident, including the independent report commissioned by OpenAI, relentlessly personified the AI “agents” without qualification, talking of them as scheming, lying, cheating, even self-sacrificing beings. Some reports described them as a “civilization”.
We lack a language adequate to describe such goal-directed behaviour in ways that don’t seem to attribute some self-aware planning and choice to that behaviour. But the industry doesn’t seem to care. It is quite happy to present these models as natural agents akin to viruses, rather than products made by engineers. The idea that they can “go rogue” has now permeated the media, again framing the engineers as struggling doggedly to contain the miscreants.
But in the end the OpenAI model could do what it did because its creators used inadequate safety measures and were unable to control their system. (All the same, they could have stopped it at any time, had they been paying attention, simply by unplugging the machine—an option oddly absent in AI doomsday scenarios.)
Warnings about AI never concern its potential to cause serious harms, but instead fixate on the end of days
In other words, the narrative again misdirects responsibility and accountability away from the makers of the technology. The “mythology” engendered by the habitual metaphors of the AI industry (and aped by the media) matters, writes linguist Brigitte Nerlich of the University of Nottingham, an expert on metaphors in science, because it “obscures something important, namely where the failure occurred that marked this and other incidents like it”.
After the Hugging Face hack, Anthropic’s cofounder Dario Amodei introduced a new weasel word into the AI lexicon: “pacing”. “We must slow the pace at which we improve the capabilities of AI models,” he said, so that they do not outstrip the development of safety measures and regulations. His comments drew support from Altman and Elon Musk, a co-founder of OpenAI. But isn’t it deeply peculiar for companies to declare that, because their products are so dangerous, they will simply make them more slowly in the hope that we can find ways to mitigate the hazards in the meantime? (Amodei also did not bother to explain why the truly dangerous AI models, as opposed to handy ones such as AlphaFold2, are so urgently needed anyway.)
As Stilgoe says, “For AI governance, ‘pacing’, ‘pausing’ and ‘slowing down’ are the wrong metaphors. [They are] only used by people who want this to be seen as a race. This is about steering, not braking.” To speak of pacing, Stilgoe points out, is to say “we need to stop ourselves being so clever”. What is really needed, according to Bender, “is not something out of science fiction”—a way to avoid malevolent, superintelligent machines. Rather, “It’s regulation, empowerment of ordinary people and empowerment of workers.”
Join the cult
Donald Trump did not take kindly to the prospect of slowing progress. He has called expressions of concern like Amodei’s a hoax and argued that the only control needed for AI is “a STRONG AND SMART (High IQ!) PRESIDENT”. But it’s not simply that Trump sees AI as an instrument of power (and as the source of his endless slop postings on Truth Social). A Trumpian ideology is entrenched in the industry.
Many vociferous AI doomers, including Bostrom, Yudkowsky and Skype founder Jaan Tallinn, are associated with effective altruism, the movement that began in the 2010s which advocated making as much money as possible, by almost whatever means, provided you then use it to better the lot of humankind in general. The disgraced cryptocurrency entrepreneur Sam Bankman-Fried was among those who embraced the former objective while seeming indefinitely to defer the latter.
Effective altruism shades into a movement called extropianism, which asserts a moral imperative to boost the intelligence and capacity for improvement in humankind by whatever means possible, overcoming any political, biological or physical barriers. In the “effective accelerationist” view—which some proponents think will culminate in a merging of human and machine, effectively making our minds immortal—AI is a crucial component of advancement. There is overlap too with the “rationalist” movement prominent in the San Francisco community, for which Yudkowsky’s blog LessWrong is a hub. As the writer Tara Isabella Burton has commented, “Central to the rationalist worldview was the idea that nothing—not social niceties, not fear of political incorrectness, certainly not unwarranted emotion—could, or should, get between human beings and their ability to apprehend the world as it really is.”
Timnit Gebru, former AI ethicist at Google, and the philosopher Émile Torres have bundled this confusing proliferation of isms into the unwieldy acronym Tescreal (transhumanism, extropianism, singularitarianism, cosmism, rationalism, effective altruism and longtermism), which describes a hard-right-leaning, cult-like libertarian community of interconnected ideologies. “Many of the very same discriminatory attitudes that animated eugenicists in the past (eg, racism, xenophobia, classism, ableism, and sexism) remain widespread within the movement to build AGI,” they say, “resulting in systems that harm marginalised groups and centralise power, while using the language of ‘safety’ and ‘benefiting humanity’ to evade accountability”.
Not all of those forecasting an existential threat from AI share this cultish Silicon Valley mindset, but it is not a fringe view. One can find elements of it in the remarks of Altman, Musk, PayPal cofounder Peter Thiel (who is “worried about the Antichrist” and wrote in 2009 that “I no longer believe that freedom and democracy are compatible”), and billionaire tech entrepreneur Marc Andreessen (“We believe in ambition, aggression, persistence, relentlessness—strength… Our enemy is the ivory tower”). Amid this bombast, it is unsurprising that the warnings about AI never concern its potential to cause serious harms to individuals, society, and democracy, but instead fixate on the end of days.
It must be understood that these are the apocalyptic currents in which minor players such as Coxon swim. Aside from some Trumpian minimisation of climate change, it’s not easy to gauge Coxon’s own view. But witness another AI employee, Bilal Chughtai, who resigned from Google DeepMind in mid-September because he “earnestly believe[s] that AI has the potential to kill us all, and that we might be running out of time”. His comments were solemnly but incuriously covered by Reuters and the Telegraph. A glimpse at Chughtai’s blog reveals a young man entranced by Tescreal tropes, quoting Thiel reverently and avidly consuming Yudkowsky’s bizarre online fan fiction, the 660,000-word Harry Potter and the Methods of Rationality (I’m not joking). If he were my son, I’d be worried about him.
Challenging the consensus
Like much else right now in the domains of wealth and influence, the Tescreal world is so crazy that you feel tainted with lunacy just writing about it. Can the grave but bureaucratic business of devising regulatory AI frameworks truly be happening against so baroque a backdrop? Yet several books, including Adam Becker’s More Everything Forever, Gil Duran’s The Nerd Reich, Jill Lepore’s The Rise and Fall of the Artificial State, Quinn Slobodian’s Hayek’s Bastards and Muskism, and Naomi Klein’s End Times Fascism (with Astra Taylor) all patiently lay out the fevered catastrophism, ecstatic techno-utopianism and explicitly fascistic and eugenicist visions swirling around Silicon Valley. Exposure carries a high risk of infection.
When Eric Schmidt, former CEO of Google, spoke of a “San Francisco consensus” on the revolutionary prospects of AI, he was inadvertently speaking the quiet part out loud. The tech community inhabits a bubble in which they all confirm one another’s peculiar view of reality, against which even Nobel prizes offer no inoculation.
But we do not have to make that view ours. We can choose to stop seeing AI developers as untouchable geniuses, saviours of humanity who build incomprehensible god-machines with magical juices flowing around their sentient silicon circuits. We can push back against their grandiosity, their evasions and fantasies, and demand that their products be safety-tested, regulated and evaluated as we do with more prosaic technologies like drugs and cars. That would be the most effective, and ultimately the safest, response to the current panic.
