This summer was filled with worries about bad AI behaviour. First, agents built by OpenAI hacked into servers belonging to Hugging Face, another AI company, after determining they might be able to cheat at their tasks by accessing clandestine data. In a detail that fascinated commentators, dozens of different AI agents communicated via a hacked message board, coordinating their efforts.
Then in September Anthropic employee Jacob Coxon resigned, citing fears that AI could soon threaten humanity if systems’ capabilities went beyond our ability to monitor and control them. Evan Hubinger, Anthropic’s head of alignment science (the art of getting AIs to behave how we want them to), agreed, warning there is a greater than 10 per cent chance of AIs eliminating us in the next decade. Some politicians and AI CEOs have since proposed a pause in development to give alignment researchers and ethicists a chance to catch up.
Is this panic another example of “critihype”, in which companies covertly advertise their systems by warning of their power? Or perhaps a stalling tactic to give Anthropic and OpenAI time to figure out viable revenue models before listing on the stock market?
Either way, and whatever the chances that machines will exterminate me before I pay off my auto loan, we should also pay attention to another piece of terrifying research. In “State media control influences large language models”, a team of US social scientists examine a range of AIs, including “closed” commercial models like ChatGPT, as well as “open” models used by researchers, including Meta’s Llama.
Focusing mainly on Chinese media, they found that large language models of all kinds routinely returned government-sponsored propaganda, especially when prompted in Chinese. Propaganda phrases were repeated by chatbots at a higher frequency than those from other sources, and so the models gave answers to questions about Chinese politics that were favourable to the Chinese government.
We might expect AIs built in China, such as DeepSeek, to produce pro-government responses because they’ve been ordered to do so. But these responses were also found in US-based models including Claude and ChatGPT. It’s not the heavy hand of government censors directly influencing AI responses, but something far more subtle and insidious.
One of the study’s lead authors, Hannah Waight, said: “People often talk about AI as if it learns from the internet in some neutral way. It doesn’t.” Instead, powerful institutions that influence the press and media environment help shape what AI knows.
The study examined corpora—vast collections of documents—used to train AI systems. It found documents from Chinese state media were far more common than any other Chinese-language source—more than ten times more prevalent than those from Chinese Wikipedia or the commercial encyclopedia from Chinese company Baidu.
Answers extrapolated from those sources tend to support points of view favoured by the Chinese government. More surprisingly, the authors found a “spillover” effect, in which languages that share “tokens”—elements of written speech—with Chinese also showed significant pro-government bias. The effect was especially noticeable in Japanese and Korean, but the authors note that biased responses occurred in English as well.
Large language models learn facts about the world from millions of documents. AI researchers discovered that their models become more capable when they ingest huge numbers of documents, rather than being fed a carefully curated selection. To build more powerful models, AI engineers have used every document available, including pirate copies of copyrighted books, transcripts of YouTube videos and, most recently, out-of-print books, bought in bulk, scanned and disposed of in data centres in Nevada.
All this encompasses an amazing range of information, from peer-reviewed academic papers and carefully researched journalism, to trolling on social media and conspiracy theory threads on Reddit. To some extent, these are all useful raw material for teaching an AI how to craft a grammatically correct sentence. (Okay, maybe not the Reddit threads.) But there’s a wide range of quality in terms of teaching an AI how to give accurate or fair responses to questions.
For common enquiries—“who won the 2020 US presidential election?”—humans have evaluated a system’s responses and steered it towards verifiable, carefully phrased responses. But for less common questions—say, the finer points of industrial planning in Hunan province—the answers are “inferences” from the documents a system has been trained on.
Beyond China, the authors found an intriguing general pattern: when you ask an AI questions in the language spoken in a less-than-open society (Vietnamese or Turkmen, for example), the system is more likely to provide pro-government responses than critical ones.
The explanation is straightforward. There is an abundance of pro-government material by state media, and a scarcity of independent critical voices. This scarcity can be enforced by offline and online threats—demanding YouTube take down a critical video or threatening activists with jail can both rig the broader information environment.
But the implications go well beyond government propaganda. Most of what AIs “know” is extrapolated from what humans know. While early AIs drew few distinctions between information from a respected newspaper and information from random bulletin boards, modern AIs check their answers against lists of believable sources.
But verification of what’s true in the world requires the expensive work of reporting, and AIs are further undercutting the advertising revenue most publications have relied on. Why follow a search engine link to a report, and view some ads in the process, when an AI will give a plausible answer on the spot?
While paying reporters to establish facts becomes even more challenging, the price of opinionated bullshit gets ever lower. In 2024, researchers in France discovered “the Pravda network”, a huge set of AI-powered news sites, creating barely comprehensible pro-Kremlin stories designed to inject perspectives into large language models and Wikipedia. It costs very little to publish waves of AI-generated content promoting a particular narrative, and vastly more to carry out the reporting necessary to counter these narratives with more neutral perspectives.
Defending democracy from AI-powered propaganda would be difficult in any time. But that battle is unfolding at a moment where credible journalistic outlets are battening down the hatches to prevent their content being hoovered up by AI training bots. The non-profit global news site I helped found—Global Voices—has had to bar traffic from whole nations when swarms of bots have so aggressively scraped content it knocked our site offline.
The theft of journalistic content is encouraging publishers to become less accessible to human readers, while enabling propagandists to flood the internet more quickly and efficiently. Regardless of whether AI becomes capable of ending humanity, we should also worry whether AI will kill off informed democracy. Unfortunately, the chances of that happening are likely to be much greater than one in ten.
