Whenever a new technology is released into the world, one group of people embraces the innovation and argues that it will transform society forever. Think of the enthusiasts who took up crypto, predicting the transformation of business, culture and society as a whole. A second group of people recoils in fear: whether they fail to understand the technology or find it actively threatening, they think nothing about the innovation can be good news.
A third group interrogates the new technology and discovers ways that it breaks and fails. These critics can seem as if they are missing the wood for the trees. But creatively breaking these new systems is an essential step if we are to ensure that a technology that promises to change everything will change it for the better.
The technology that is currently predicted to change everything is artificial intelligence built around the “large language model”, meaning it learns patterns from billions of online documents. One notable example is ChatGPT, developed by start-up OpenAI. ChatGPT simply predicts which words go together in plausible sentences. At the most basic level, all it is doing is putting one word after another. Then again, that’s all Homer was doing in the Iliad. Putting one word after another can be powerful if those words reveal new and unexpected things about the world.
Text from ChatGPT often feels as if it was created by a human and can be very sophisticated: writers have asked for Shakespearean sonnets about organic chemistry and got extremely creative verse in return. It can also be practical: ask ChatGPT to provide an introduction to a field of knowledge and you will likely get a helpful response. Models like ChatGPT can sometimes produce executable computer code, and students are finding they can fool teachers with ChatGPT-generated essays.
The power of such models has been embraced to solve a problem that is increasingly apparent on the internet: search is broken. If you go to your search engine of choice and ask it to “recommend a hotel in the fourth arrondissement of Paris that costs less than €200 a night and is an easy walk from public transport”, you’ll likely get results from businesses that compete to sell reservations: Expedia, TripAdvisor, Agoda. They generally aren’t addressing your specific query—they might narrow to a neighbourhood, but a multi-parameter search would require a user to page through many results to sort the information into a recommendation. However, if you pose the question to a large language model-enabled search tool—say, the chat facility available with Microsoft’s Bing, which began the arms race in this field—it is powerful enough to sort the information itself and give you a thoughtful list in accordance with your criteria. Marketers haven’t yet been able to “game” the system in quite the same way by inundating users with promotional garbage (though it’s easy to imagine advertisers paying to be seamlessly recommended in this way in future).
What’s not clear is whether the results will be accurate or unbiased. Testers of Bing’s chat-enabled search have reported that the system can recommend events that happened in the past, or restaurants that have closed. In one example during a demonstration, the tool mangled reports of a firm’s financial records.
Bing itself now warns that its AI-generated answers will “sometimes misrepresent the information it finds, and you may see responses that sound convincing but are incomplete, inaccurate, or inappropriate. Use your own judgment and double check the facts before making decisions or taking action based on Bing’s responses.” Good advice for any information online, but hardly a ringing endorsement.
The ways large language models get things wrong can be fascinating. These systems are optimised for plausibility, not accuracy. If you ask for academic writing, it will output text complete with footnotes, because the writing the response is modelled on contains footnotes. But search for the papers cited and you may discover that they do not exist, as librarians have found when dismayed students seek their help in locating papers that ChatGPT has simply invented.
Scholars of machine learning refer to these errors as “hallucinations”: the system has dreamed up footnotes to make its text look more convincing. If you want to see a large language model hallucinate, ask it about a subject you know very well: yourself. I found that, no matter how many times I asked the system to generate my biography, it would make certain errors over and over. Notably, ChatGPT wants me to have a more impressive resume. When I generated a bio, it would give me a master’s degree, usually from Harvard, or sometimes a PhD from MIT. Those are both plausible errors, as I’ve taught at both universities. However, I only have a bachelor’s degree.
ChatGPT wants me to have a more impressive resume
I’ve begun to think of this phenomenon as an “upgrade”. I asked a group of friends—some notable academics—to try the experiment as well. ChatGPT granted some of them awards they had not won, such as the MacArthur “genius grant”. Several friends who teach computer science had won Nobel prizes—impressive, given the fact that there is no Nobel awarded in this subject.
Other upgrades involved our personal lives. Virtually everyone’s biography included a spouse and two children, including cases where the subject was single. I interrogated ChatGPT about the bio it produced of a friend—recently divorced—whom it had described as married with two children.
It responded, “Yes, further checking my records, I can see she announced on her blog in 2019 that she and her husband divorced.” In other words, ChatGPT both “knows” my friend is divorced, and that she is married with two children. In truth, ChatGPT doesn’t know anything about my friend. It is simply constructing plausible sentences based on the massive collection of information it has absorbed from the web. Biographies online tend to be written about impressive people, the sorts of people with prestigious degrees and multiple awards. People who mention their family status tend to be happy about the state of their families. It is rare that you will see someone’s bio explain that they live alone with a disinterested cat. Thus, the bios ChatGPT generates encode these biases in the process of telling plausible stories of lives lived.
(Amusingly, the other thing my friends noticed was a tendency for ChatGPT to bump them off; some reported glowing biographies that ended with their untimely deaths. Again, this is an explainable bias: many biographies appear in the form of obituaries.)
Documenting the false assumptions of these systems can feel like nit-picking, but it’s important work. Researchers investigating early versions of image-generating AIs found blatant racial and gender biases: ask for a picture of a doctor, and systems were likely to generate a white man in a lab coat. Worse, asking for an image of “a man sitting in a prison cell” would disproportionately generate images of men of colour. Rather than companies doing the hard work of ensuring a less biased set of images to train their systems on, Richard Zhang of Adobe Research found that OpenAI’s image-generation tool seemed to be covertly adding the terms “black” or “female” to searches to get more diverse results. (Zhang cleverly asked DALL-E to produce “a person holding a sign that says”, deliberately leaving the sentence unfinished. He received results of a black woman holding a sign that said “black”.) Such crude attempts to mitigate bias are better than nothing, but their main effect is to mask bias, making true progress harder.
I am part of a group of critics who initially dismissed the power of these new AIs and are now eating our words. In a column for this magazine last year, I predicted that systems like DALL-E would not take jobs away from human artists, but instead be tools for co-creation. Yet having seen reputable publications use image auto-generation to illustrate stories, I feel far less confident.
ChatGPT may well make search systems better. But it is essential that we interrogate such tools, to avoid inadvertently reinforcing biases in the process of adopting a new technology whose powers can seem almost magical.
