It is important to resist the temptation to cherry-pick polls that support one’s caseby Peter Kellner / December 18, 2017 / Leave a comment
Sorry, Remainers. The BMG poll for the Independent, which caused such a stir over the weekend, looks wrong. The Remain lead, which has hovered around 1-4 per cent in recent weeks, has NOT suddenly widened to ten points in the wake of the government’s embarrassing defeat last week in the House of Commons.
Here are four reasons to be wary of BMG’s figures.
First, they are mainly driven by a seemingly huge shift in people who did not vote in last year’s referendum. Sure, the Remain camp is being swelled by young adults who abstained last time or were not old enough to vote. But previous BMG polls included this phenomenon.
During the summer, when previous non-voters were asked how they would vote in a fresh referendum, they divided fairly steadily: around 45 per cent Remain, 25 per cent Leave. Now, suddenly, BMG say the divide is 67 per cent—16 per cent. As this group comprises more than a fifth of BMG’s total weighted sample (299 out of 1,363), this 51-point Remain lead within this group accounts the reported overall Remain lead.
However, BMG did not actually interview 299 previous non-voters. Its unweighted subsample was barely half that: 156. The margin of error on such a subsample is large—and the very fact that BMG could not track down as many non-voters as it wanted, provides a clear warning (as I know from my experience at YouGov) that the sub-sample may not be as representative as one would like.
Secondly, if it were true that, despite these sampling issues, there had been a sharp shift among non-voters to Remain, there would be some echo of this among other groups. In particular, one would expect to see clear signs of growing buyers’ remorse among Leave voters.
That wouldn’t apply to hardline anti-EU voters—something like two thirds of those who voted Leave 18 months ago. But it would be likely to have some effect on the one-third who were “instrumental” voters: people who are not viscerally anti-EU, but believed that Brexit held the best hope of more jobs, higher pay, less crime, a better-funded NHS and improved access to public services such as local schools and social housing.
The point is that any weakening of these pro-Brexit arguments that is liable to shift large numbers of non-voters into the Remain camp, should also produce some shift among the “instrumental” Leave voters. But BMG’s figures produce no evidence of this. There is no statistically significant rise in buyers’ remorse among Leave voters in this poll compared with previous BMG polls.
The third reason to doubt that BMG’s figures reflect a public reaction to last week’s government defeat in the House of Commons is that the poll was conducted before the vote. Its fieldwork dates were 5th – 8th December—that is the week before last.
Fourthly, other polls within the past fortnight show no indication of a lurch to Remain. Two YouGov surveys and one ICM poll indicate that nothing much has changed in recent weeks, despite the turbulence surrounding the events leading to the EU’s decision to allow Brexit negotiations to move to stage two.
They are consistent with a narrow Remain lead, compared with a narrow Leave lead before this year’s general election. Over the past six months, as I discussed in a recent blog, there has been a small shift to Remain, but only a small shift.
To say this is not in any way to denigrate the quality of BMG’s work. Small subsamples of hard-to-reach groups must always be examined with care, for they are liable to trip up even the best research companies. BMG seem to have been unlucky, not culpable.
The larger point is one that applies to enthusiasts on any side of any issue. It is important to resist the temptation to cherry-pick those polls and findings that support one’s case, and ignore those that don’t. Remember the warning of those great 20th century philosophers, Simon and Garfunkel, in “The Boxer”: “All lies and jest: a man hears what he wants to hear, and disregards the rest.”
Brexit Britain: the future of industry is a publication which examines the future of UK manufacturing through the prism of the recently released Industrial Strategy White Paper. The report features contributions from the likes of Greg Clark MP, Miriam Gonzalez, Richard Graham MP and Frances O’Grady.
To find out more about how you can become involved in Prospect’s thought leadership programmes, please contact firstname.lastname@example.org.
If you want to know all about where industry is headed in Brexit Britain, you can download the whole Brexit Britain: The future of industry reportas a fully designed PDF document. To do so, simply enter your email below. You’ll receive your copy completely free—within minutes.
When you sign up for this free report, you will also join our free Prospect newsletter.
Prospect takes your privacy seriously. We promise never to rent or sell your e-mail address to any third party. You can unsubscribe from the Prospect newsletter at any time.