Poll results vary widely depending on how the question is framedby Peter Kellner / October 19, 2011 / Leave a comment
The evidence that I am going to give you now will provide ammunition for those who don’t like opinion polls, think the numbers mean little and reckon our political culture would be healthier without them.
Of course, that’s not how I view this evidence. I believe it shows the need to read poll results more carefully. We need to distinguish between top-of-the-head answers to questions asked out of the blue, and how people react once they have thought about the matter in hand.
Here’s what we did. To inaugurate a new partnership between YouGov and the department of politics at Cambridge University, we tested how the wording and context of a question can shape responses. We all know how results can be biased by loaded questions, such as “Do you think Britain should allow the tinpot dictators in Brussels to tell us what to do?” So we asked alternative, unbiased questions. We chose the BBC licence fee: do people regard it as good or bad value? We posed the question in nine different ways, with nine different samples of more than 2,000 people, in late August and September.
The graphs overleaf show what we found. When we asked the simplest question: “Overall, do you think the BBC licence fee is good or bad value for money?” we found that equal numbers of people replied “good” and “bad.” This yields a net score (good minus bad) of zero.
But when we started adding information, the figures changed. Reminded that the licence fee costs £145.50 a year, people told us by two-to-one that it was bad value—a net score of minus 27. When we divided the annual sum into smaller timescales, responses became steadily more positive. If people are told the licence fee works out at 40p per day, the net score is plus 8.
Next, we added in the cost of a Sky package, and quoted figures (as Sky does) in terms of monthly subscriptions. When told a basic Sky package costs £19.50 and the licence fee £12.13, people give Sky a positive net score, and the BBC a negative score. But when we mentioned that a premium Sky package (including sports and movies) costs £55.75 a month, the BBC’s net score turned positive—and Sky’s turned negative.
Finally, we asked a warm-up question: how people feel about the service provided by the BBC. Three quarters of the public say they are…