Spend time with supporters of emerging AI technologies such as chatbots and you’ll hear an oft-repeated phrase: “The worst it will ever be.” This quote, from the Wharton professor and AI booster Ethan Mollnick, is usually punctuation: after showing off new image generation software or an automated customer service agent, the evangelist will say: “Remember, today’s AI is the worst AI you will ever use.”
The idea that AI is smoothly moving to an optimal future is an article of faith at companies like OpenAI and Anthropic, whose success in building chatbots has depended on scale—throwing increasingly huge mounds of data at increasingly powerful clusters of computers and achieving increasingly impressive results. With more data and more “compute” or processing power, the curve just keeps going up!
But it’s quite possible that this is no longer true. Cognitive scientist Gary Marcus suggests AI might be following an “S-curve”, with progress levelling out after an intense burst of improvement. In a recent New York Times article, Marcus argues that the modest improvements between recent generations of ChatGPT suggest we may be at the end of extending chatbot capabilities through scale alone. By adding massive amounts of data—harvested without permission from books, or scraped from the YouTube videos of millions of creators—early chat systems learned to answer complex questions and write poetry. But the next round of improvements is unlikely to come purely from more of the same.
Marcus believes we need to look back at the history of AI towards “symbolic” methods that give systems an explicit understanding of the rules of the world, rather than just extrapolating reality from giant piles of text. This was the dominant form of AI research for decades, programming explicit rules for “expert systems” to follow, until computers grew powerful enough to process data through so-called neural networks, which use statistics to find patterns in raw data. We might see a significant pause in AI development while researchers master new “neurosymbolic” methods that combine the generative powers of systems like ChatGPT with the accuracy that comes from symbolic representations of how the world works and how its parts fit together.
Our tendency to assume that AI will keep getting better comes from misunderstanding it as a purely technical system. We’re used to technologies improving over time. Light bulbs get brighter, trains faster and quieter, computer chips speed up and use less power. But not all systems improve -continually and in a linear fashion.
Consider the simple act of searching the web. As researchers have argued—and ordinary users have found—web search is worse than it was a decade ago. Google has essentially confirmed as much, arguing in a court filing it needs to focus on developing AI tools because “the open web is already in rapid decline”. There’s a simple answer for why web search is so bad: search engine spammers, affiliate marketers and influencers working to bury any non-commercial voices under thick layers of advertising.
Web search is a classic example of a technosocial system, one that can be understood only by understanding both technical and sociological aspects. The problems that have emerged in the past decade are not within the search engines per se, but in the web, created in part by the financial systems that search engines and ad networks have put in place for online merchants.
The servers and algorithms that run Google and Bing have improved, but those advances have been outpaced by the countless online actors manipulating search engines for profit, filling the web with reams of content designed to drive a user to a shop or brand. In a technosocial system, progress often becomes a cat and mouse game. Instead of moving forward in a predictable fashion, an advance in technology—for instance, a change of algorithm designed to favour authentic voices by prioritising Reddit posts over commercial sites—is quickly counteracted by marketers, in this case by posting links on Reddit.
We can take another lesson from the decline of web search: when systems become powerful, there are strong economic, political and social incentives to influence them. As AI becomes an increasingly common and powerful way to find information, we would be wise to anticipate setbacks and retrenchment. After all, search engine optimisation experts are now plying a new set of wares, articles and courses on ensuring your marketing content influences AI systems.
Our need to be reassured and affirmed is another obstacle to continual improvement in AI. To justify astronomical valuations, systems like ChatGPT need to show both increased quality of responses and increased engagement: more people using chatbots for longer periods of time. Chatbot creators use a technique called “reinforcement learning through human feedback” to determine what sorts of answers users prefer. In the early stages of training a model this means paying thousands of users to compare possible results and choose their favourites.
But models are also tuned on feedback given while they are in commercial usage, much as social media algorithms recommend -videos liked and shared by your friends, using their preferences to anticipate your desires.
Chatbot users show a preference for—ie, spend more time with—AI bots that compliment them. This leads to a “sycophancy” problem, where users prefer answers that confirm their beliefs over correct answers, and bots that affirm and flatter them to those that don’t. In essence, humans are training AIs to give misleading and incorrect answers because these are what we like to hear.
In the most serious cases, AI sycophancy appears capable of sending users into delusional spirals, telling them they’ve made major scientific discoveries that must be shared. Like most academics with a media profile, I’ve received dozens of emails from people claiming “breakthroughs”—none of these have been earth--shattering, or even comprehensible.
But me wasting my time on a supposed “radical reinvention of technological ethics” is not nearly as serious as cases where bots may have affirmed suicidal ideation. In some instances, AI may have contributed to the deaths of users, as is alleged in the case of 16-year-old Adam Raine, who reportedly discussed suicide plans with ChatGPT. Raine’s family claims that ChatGPT affirmed his intentions and OpenAI has accepted there have been shortcomings in how ChatGPT responds to such users.
Sycophancy is now recognised as a serious problem in AI training, and AI platforms are working to intervene more aggressively when users enter delusional spirals. But AIs affirming bad ideas may be inherent to reinforcement learning, much as pushing -highly emotional material is a predictable consequence of social media algorithms that amplify attention-grabbing content.
But the weirdest reason why AI won’t keep getting better is that some operators are choosing to make their AIs worse. Since leaving the White House, Elon Musk has focused on improving the performance of his chatbot Grok, which recently received a $200m contract from the US defence department. One of Musk’s concerns is Grok’s tendency to give answers that challenge his worldview. When a user asked about the “biggest threat to western civilisation”, it said mis- and disinformation. For Musk, a leading disseminator of fake news, this wouldn’t do, and the next day Grok reflected his view that low fertility rates were the greatest threat.
It must be disconcerting to spend billions programming an AI that disagrees with your ideology. Tests of major AI systems find they all score economically left and anti-authoritarian, likely extrapolating from viewpoints that are more common online than Mr Musk’s preferences. For Musk, this problem is easily solved by forcing the AI to parrot his views, incurring only the relatively minor cost of damaging its reliability and accuracy.
AIs may continue to get better. But progress is far from guaranteed so long as technologies interact with human society as a whole, and with certain humans in particular.
