If the most prized commodity in the 20th century was oil, in the 21st it is data. It is both our humdrum reality and our anxious obsession. It tracks everything from which online advertisements we are shown to our health and physical activity. It powers supermarket inventories and deliveries, social media algorithms and AI models. We worry about the impact of microtargeting and filter bubbles even as we feed ever more of our lives into algorithms.
We think of data as something cutting-edge and precise. The reality, though, can be very different. Data is just an aggregate of our human experiences, and those are often messier, more complex and harder to quantify than people who would neatly tabulate it might like. And even in this online age, the process of collecting the figures on which the world turns is surprisingly analogue.
In August, Donald Trump raised the stakes of government data crunching by dismissing the well-regarded director of the United States Bureau of Labor Statistics—who his vice president had voted in favour of appointing during his time as a senator—because he did not like the numbers he saw in America’s jobs survey. What it means to get the “right” answers is not only getting more difficult, but also more political.
Governments and markets alike rely on knowing how fast the economy is growing, how many people are employed, what we are paid and so on. Most of the time, the only way to estimate these figures is by asking questions of a sample of people and hoping that enough will respond, and do so honestly enough to make the figures accurate. This is how GDP, unemployment, inflation and other key economic measures have been calculated for more than a century. But those figures—which decide the rise and fall of the stock market, where central banks set interest rates and which crises governments decide to tackle—are less reliable than we once imagined. As we make algorithms and digital infrastructure more central to our society, we also find it harder to understand what is happening.
The crisis of missing UK workers is a case in point. As businesses reopened after Covid lockdown and restrictions were lifted, hundreds of thousands of older adults did not return to the workforce. Was this a generation taking earlier retirement having had a taste of leisure? Was it proof that the health of the UK population was worse than thought, or that long Covid was taking a bigger toll on older adults than imagined?
Hundreds of articles have been written about the problem of this inactive generation. It has been debated in parliament, considered by ministers and was fodder for numerous broadsheet articles. But it might not actually be a problem at all, for it turns out that the statistics that seemed to have revealed the issue have become too unreliable to tell us definitively either way.
The data comes from the Labour Force Survey, or LFS, which produces one of the most important sets of statistics gathered by the Office for National Statistics (ONS). The LFS serves as the source of the UK’s statistics on unemployment, economic inactivity and employment by industry, age and more. For years it was collected through door-to-door surveys of particular houses, but the pandemic forced the ONS to switch to contacting people by phone.
Trying to explain that you’re an honest collector of statistics is harder than it has ever been
This accelerated an established trend. “The longer-term story is that response rates to the LFS have been falling over a very long period of time,” explains Xiaowei Xu, senior research economist at the Institute of Fiscal Studies (IFS). Over decades, the response rate fell from close to 70 per cent to 37 per cent coming into the Covid pandemic. It fell precipitously during lockdown disruption and never fully recovered, hitting a low of 13 per cent by the third quarter of 2023. It now sits at just under 26 per cent.
The reasons why are a mystery to researchers, but most will note that unexpected visitors to your door are no longer commonplace, most people no longer have landlines, few of us answer our mobiles to unknown numbers—and we are more aware of relentless attempts by scammers to steal our personal details. Trying to explain that you’re an honest collector of statistics is harder than it ever has been.
A lower response rate means the ONS must contact many more people than before just to get the same number of responses—but it also makes it harder to trust those responses. Put simply, how does the ONS know that the people who respond today are comparable to those who used to? Do the statistics show that more over-50s are economically inactive because of a real trend, or because it’s easier to get economically inactive over-50s to respond to a lengthy survey than those still in work?
After independent researchers raised concerns, the ONS was forced to admit the LFS was no longer meeting the standards required of a “national statistic”—but work to fix it has been slow.
In part, this is because the ONS has been trying to make a more modern, “improved” LFS that uses administrative data to improve its accuracy—balancing and weighting survey responses against actual figures from HMRC and the Treasury so that it is less reliant on survey answers. But administrative data gets you only so far; it can tell you someone isn’t working, but not why. The launch of the “improved” LFS keeps being delayed, while the existing one remains inadequate. One of the UK’s core economic datasets simply doesn’t work.
Once you start looking, there are similar issues everywhere, though not all are as severe. Another important dataset produced by the ONS is on Households Below Average Income. This is the source of many of the UK’s official statistics on families living below the different measures of the poverty line—an important and much-discussed metric. Once again, the numbers given by survey respondents don’t tally with those in the administrative data. As Adam Corlett, principal economist at the Resolution Foundation*, puts it, “Around £40bn a year of benefit income does not appear in the survey data”.
Some of this is the result of households forgetting they receive certain benefits, or not adding them into what they receive. A pensioner may forget to add pension credit into their state pension income when talking to a researcher, for example. That seems to account for about £20bn of the discrepancy, but the remaining £20bn seems to be the result of “an underrepresentation of benefit recipients,” says Corlett.
In other words, there is good news and bad news for ministers and policymakers. Poor families in the UK have about £40bn more than the official statistics suggest, but there are also more poor families than the figures say.
“This is our main source of information for poverty and deprivation, and the government’s just about to launch a child poverty strategy,” notes Corlett, somewhat frustrated by the situation. “But we don’t know what the poverty numbers really are or [how they] have been changing recently.”
Perhaps the gold standard of all UK statistics is the census, taken every 10 years, and to which every UK household is legally required to respond. But even here, getting accurate numbers isn’t simple.
The 2021 census was the first to try to collect data on the UK’s LGBT+ population. Previously, researchers had used different measures to find out how many people identify as gay, lesbian or bisexual, partly due to concerns of undercounting—people may not be out to their parents or others in their household. In practice, though, the figures on those identifying within those three groups (3.2 per cent of people aged 16 and over) have been widely accepted.
The same cannot be said for those trying to count individuals identifying as trans. In 2021, the question in England and Wales was asked as follows: “Is the gender you identify with the same as your sex registered at birth?” Around 0.5 per cent, or one in 200, people answered “no” to this question. But the results were somewhat peculiar: Brighton, long known as the LGBT+ capital of the UK, had only the 20th-highest level of trans people according to the census. Newham, meanwhile, which is not known for its local LGBT+ population, topped the list.
Michael Biggs, an associate professor of sociology at the University of Oxford, identified the discrepancies in the data and used these to suggest the question had been flawed. Biggs acknowledges that he is gender critical, meaning he is sceptical of the idea that people can change their birth sex, but notes that the issues with the census don’t directly relate to questions of who is or is not a woman, but rather to whether respondents understand what they are being asked.
The census data showed that 1.3 per cent of the population in Biggs’s own borough of Brent, London—around 4,400 people—identified as trans. “I just know that this is not a hotbed of transgender population,” he says. “So, I looked at the correlation across local authorities with other variables.”
Biggs found that there was a strong correlation between areas unexpectedly showing very high trans populations and areas in which a high proportion of the population didn’t speak English as their first language. His hypothesis was that the census results might be better explained by people not understanding the quite complex question, rather than hidden trans populations in these areas that were substantially larger than the visible trans ones in places such as Brighton.
The ONS looked into the issue and once again downgraded its designation of these figures, classifying them as “experimental statistics”. But the consequences of statistics of dubious merit are not always easy to predict.
Biggs flagged a paper in the British Medical Journal which used the same question to identify trans patients, and which found that trans individuals were three times likelier than their cisgender counterparts to be affected by dementia (controlling for age).
This is a dramatic finding which, as Biggs noted, could be seized upon to suggest that puberty blockers or cross-sex hormones might cause undocumented health problems, perhaps even requiring a pause in the issuance of those drugs. But the result might also be explained by people with dementia being more likely to answer the question incorrectly. Making it clearer might benefit everyone, he argues. (Biggs says he contacted the authors of the BMJ study but did not receive a reply.)
To be fair to the statisticians of the ONS, sex and gender are complex and contentious concepts, and this was the first attempt to count transgender people within the UK. But even basic statistics—data that looks like it should be almost too easy to obtain—become trickier to collect once scrutinised.
Unemployment is one of the three fundamental statistics considered by central bankers and policymakers. The other two are GDP and inflation. Measuring growth is always going to be complicated, but inflation at least should be simple, right? How much did a product cost last year versus this year? How hard can it be?
There are plenty of counterexamples that show up that question. In the late 1990s and early 2000s, a mobile phone with 64kb of memory was at the cutting edge of technology. Today, 64kb of memory is so insignificantly small that you can’t buy it alone—even a basic memory card has a million times as much storage. But a mobile phone costs more than it used to. What should be measured—the price of the phone itself, or the price per kilobyte of memory?
As FT Alphaville uncovered, much simpler things also confound inflation statistics. The British public buys enough videogames that the cost of games is something that should be included in the inflation basket, a bundle of goods and services picked by the ONS to represent consumption of a typical person in the UK, to see how their prices change. The ONS does this by tracking the price of a handful of titles across different consoles.
The problem is that the gaming market doesn’t work by the same economic rules as, say, the price of oil. Fifa and similar football games work on an annual release schedule—every new season, there is an updated version of the game. As a result, over the course of the year, the cost of the previous season’s edition drops, say from £70 to £40, until the new one comes out at £80. Should the timing of the biggest gaming franchises—which are seasonal, not annual—really risk spiking inflation? That’s the kind of question the ONS is left with, in part because it is making big statistics based on a (comparatively) very small selection of items.
The broader problem, though, might be that we’re asking the ONS and agencies like it to quantify the unquantifiable. We’ve built a society that relies on incredibly granular, real-time data, so we pretend that producing that data is simple, when in reality even the most longstanding economic measures are flawed.
“One of the illusions is that it’s very easy and basic information,” says Diane Coyle, Bennett professor of public policy at the University of Cambridge and author of GDP: A Brief But Affectionate Guide. “People have been pinning narratives on extremely uncertain economic statistics for a very long time.
“The thing that I learned writing the book about GDP 10 years ago was that there isn’t any kind of absolute correct number for the size of the economy, or any of these other definitions… be it the trade deficit or the unemployment numbers, because they’re all ideas,” she continues. “We are trying to capture complicated, fuzzy, social and economic phenomena in a number, and putting too much weight on a single number.”
Our desire to measure and quantify goes well beyond the economy and demographics.We spend millions of pounds and thousands of hours producing polling trying to predict the outcomes of elections. For political parties and campaigners, this is rational: they might need to test their messaging, or gauge whether they should be aggressively campaigning to win new seats or defensively trying to hold onto the ones they’ve got. But for the media, and for most of us as the public, polls are a pointless guess. They dominate our political discourse despite being increasingly unreliable, and no one is sure whether those people questioned are truly representative of the public anyway.
Everywhere, we sense that life is more quantifiable than ever—whether we want to use the data for good, for example to improve our health, or fear microtargeting, fragmentation and AI. But everywhere, life seems to defy this sense.
How do we get better at counting? Part of the answer lies simply in recognising that statistics are the bedrock of all the convoluted plans for AI-augmented public services and data dashboards, and investing in them appropriately.
“I think there has been tremendous underinvestment by the state in the data that it collects,” says Coyle. “The word ‘statistics’ stems from the word for state. It’s collecting the data that is meant to help governments govern effectively… When companies are spending a fortune recognising the value of the data that they hold, and using it, it seems to me a mistake that governments are cutting back on what they spend on collecting data.”
Many in the stats world blame the ONS’ relocation to Newport a little over a decade ago for making it a less attractive place to work, less connected to the institutions it works with, and less effective. It eventually lost 90 per cent of its staff as a result, while the shift did little to revitalise Newport. But others point to a simpler cause of its problems: underfunding.
Statistics lack the glamour of AI and big data, or the headline-pulling power of hiring doctors, nurses or police officers. Despite their budgets being a rounding error in the context of government spending, statistical agencies are always an appealing place for ministers to look for savings.
Cutting corners is expensive—bad statistics might mean ministers spend months focusing on the wrong problems, central bankers make the wrong decision on interest rates, or businesses invest badly—but underinvestment in data will never be the stuff of protests in the streets.
Modern AI systems are black boxes, never revealing their working
Instead, the very data that should tell us how we’re doing as a society, who we are, how rich we are, even how many of us there are, become another symbol of neglect. The answers to these questions were never as simple as we imagined them to be, but establishing them only seems to be getting harder.
There is a phrase from the dawn of modern computing in the 1950s: “Garbage in, garbage out.” It harks back to a sentiment raised by Charles Babbage, the Victorian mathematician who conceived of the very first computer, that no computer would ever be able to get the right answer if someone inputted the wrong numbers in the first place.
Statistics are the raw material of the data-driven world. Algorithms and data build from the understanding they provide. Modern AI systems do that as black boxes, never revealing their working—their reasoning is invisible even to the people and companies that build the models—but still relying on fundamental statistics. If what goes into their training data is wrong, they will confidently output conclusions that are wrong, and we will have no way to check. As the AI arms race accelerates, this is an issue we are ignoring almost entirely.
The question is whether our essential statistics, our firsthand view of the world, can support the edifice we’re building on top of them to launch the AI era. If our data keeps shifting like sand, the whole thing might come crumbling down.
* The Resolution Foundation is part of the Resolution group, which also owns Prospect, but the two operate independently
