Polling Observatory Scottish referendum special: who is ahead and how close is it?

DipticBy The Polling Observatory (Robert FordWill JenningsMark Pickup and Christopher Wlezien). The homepage of The Polling Observatory can be found here, and you can read more posts by The Polling Observatory here.

This is a Scottish independence special of our regular series of posts that reports on the state of support for the parties in Westminster as measured by opinion polls. By pooling together all the available polling evidence we can reduce the impact of the random variation each individual survey inevitably produces. Most of the short term advances and setbacks in the polls are nothing more than random noise; the underlying trends – in which we are interested and which best assess the state of public opinion – are relatively stable and little influenced by day-to-day events. Further details of the method we use to build our estimates of public opinion can be found here.

In recent weeks the debate over Scottish independence has reached fever-pitch, and debate over some of the polls has been just as fierce. Most notably a YouGov poll for the Sunday Times, published on September 7th, caused shock waves both North of the border and in Westminster when it showed Yes marginally ahead, the first lead for the “yes” campaign in many months.

As regular readers of the Polling Observatory will know, we tend to take a conservative view of sudden movements in public opinion. Most short-term shifts in the polls are nothing more than random noise. Indeed, a recent study shows that apparent swings in public opinion during campaigns are often not due to actual shifts in opinion, but instead result from differential response rates: in other words, supporters of side currently faring worse in the polls are less likely to respond when surveyed.

The method we apply to Westminster polling can also be used to look at the underlying balance of opinion in Scotland. We estimate current referendum sentiment by pooling all the currently available polling data, while taking into account the estimated biases of the individual pollsters (“house effects”). Because we have no fixed reference point for “true” public opinion (i.e. the accuracy of all the pollsters will only become known on the day of the vote), we make adjustments relative to the “average pollster”. This assumes that the polling industry as a whole will not be biased. This assumption could prove wrong, of course but we have no way of knowing what any biases might be. We can only draw on the historical record – and referendums are both very rare and very unusual political events, so this record must be interpreted with caution. That important caveat noted, it is worth highlighting that past evidence suggests an underestimation of No support is somewhat more likely.Scottish without dk 08-09-14 (1)

Our data cover the period up 6th September, including the YouGov poll that put the Yes camp in the lead for the first time (so does not include subsequent polls by Survation and YouGov that have the No camp back in the lead). Support for the Yes camp has surged in the last six weeks or so, leading some forecasts to put the chances at close to even, but the independence campaign still lagged in our estimates even after the YouGov poll. Subsequent polls, including one on September 11th by YouGov, have all shown No narrowly ahead, suggesting the 48%-52% margin we find here has not altered in the last few days. While the race has got considerably closer, the polling evidence still makes No the favourite.

The same pattern is repeated if we look at the trend for unadjusted responses below (undecided voters are not plotted on our graph), although here support for the Yes and No camps is even closer. There is some suggestion here that Yes has surged by winning over “don’t knows” rather than converting No supporters – Yes has risen by a good deal more than No has dropped. However, the current balance of opinion suggests that Yes will either have to win a large majority of the remaining undecided or convert some No votes to its camp in the final few days.

Scottish with dk 08-09-14 (1)

As mentioned above, our method also makes it possible to estimate the “house effect” for the survey responses for each polling company, relative to the Yes and No figures we would expect from the average pollster. That is, it tells us simply whether the reported share of responses for Yes, No and Don’t Knows is above or below the industry average. This does not indicate “accuracy”, since there is no final vote against which to benchmark the accuracy of the polls. It could be, in fact, that pollsters at one end of the extreme or the other are giving a more accurate picture of voters’ intentions – but the final vote is the only real test, and even that is imperfect. In the table below, we report all companies’ support for the Yes and No camps (and for don’t knows) relative to the median pollster.

Table: House effects of independence pollsters relative to median pollster

House Mode Yes No DK
YouGov Internet -0.5 4.6 -4.2
ICM Internet 0.5 -2.1 1.5
Panelbase Internet 4.4 -2.3 -1.9
Ipsos-MORI Telephone -0.6 6.5 -5.9
TNS-BRMB Face-to-face -5.6 -4.2 9.8
Survation Internet 1.5 -0.1 -1.0

Our estimates show substantial differences across polling houses. While YouGov (+4.6) and Ipsos-MORI (+6.5) tend to report higher responses for the No camp, Panelbase (+4.4) and Survation (+1.5) tend to show higher numbers for the Yes camp. Interestingly, TNS-BRMB, which is the only polling company to carry out face-to-face surveys about the referendum, report lower responses for both the Yes and No camps, and a much higher proportion of undecided voters.

With just over a week now until Scotland goes to the polls, and with the campaign raging on as big hitters on both sides of the border fight it out, there is still time for late movements in public opinion. A lot will also depend on the turnout – voter registration for the referendum is exceptionally high at over 97% – well above the 85% registration rates typically seen in other elections. Figuring out how these less politically engaged voters will behave, and how many will cast a ballot, is an unusual challenge for both pollsters and the campaigns themselves.

One thing we know for certain is that opinion is closely divided. We should therefore exercise even more caution when digesting the latest shift in the polls from Yes to No or No to Yes. Everyone engaged with this passionate debate will be keen to read their own preferences into every tremor of public opinion, which means now more than ever we must keep in mind than many of the small moves in coming days will be nothing more than the random fluctuations in patterns of response that are an inherent part of opinion polling. When the result is tight enough to be within the margin of error, polls showing Yes at 49% and 51% amount to the same thing – it’s “squeaky bum time”.

Robert FordWill JenningsMark Pickup and Christopher Wlezien

10 thoughts on “Polling Observatory Scottish referendum special: who is ahead and how close is it?

  1. The main source of uncertainty is not in the margin for error in the polls, but in whether there will be differential turnouts. For the Yes side to convert its support into actual votes requires a large number of voters to vote who have never voted before, and these people are hard to convert, and to get a large number of people who have only ever voted Labour to vote against habit. These barriers are far less significant for the No side.

  2. Virtually everything else you write is totally undermined by this throwaway phrase:
    “the final vote is the only real test, and even that is imperfect”

    This is such a giveaway – revealing writers for whom the reality of politics & of people’s lives is of secondary importance to some absurd Platonic ideal of polling.

    Here’s how it actually is, chaps – the poll in which some 85% of people living in Scotland will vote next Thursday is real. Everything else is simulation. It might be a good simulation. Or it might be a bad simulation. But it’s not real.

    And to somehow contrive to compare reality with the greater or lesser degrees of “perfection” of your simulations, just shows how cut off from reality you folks have got.

    Turns out pretty much the same thing is happening with all these terribly clever economic modellers who turn out to know nothing of the reality of real life economics.

    • You completely misunderstand what we say. The point is that it is not possible to fully assess the accuracy of different polling methodologies, even against the final vote, because of sampling error (i.e. the most accurate pollster may just get lucky due to random error). So we are not saying polling is perfect, and completely the opposite.

  3. […] Finally, previous poll reports showed tendencies analyzing 87 surveys from February 2013 to September 2014 using the question wording: “Should Scotland be an independent country?” here http://whatscotlandthinks.org/questions/should-scotland-be-an-independent-country-1#line and a second analysis available here https://sotonpolitics.org/2014/09/12/polling-observatory-scottish-referendum-special-who-is-ahead-and… […]

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s