The Polling Observatory Forecast #1: Lessons for 2015 from polling history

With a year to go, the Polling Observatory team launch their forecast for the 2015 general election…

Starting this month, the Polling Observatory team is joined by a new member: our old friend and colleague, Christopher Wlezien of the University of Texas at Austin, who will be helping us to produce a long term forecast for the 2015 General Election, using methods we first applied ahead of the 2010 election. Our method involves trying to make the best use of past polling evidence as a guide to forecast the likeliest support levels for each party in next May’s election, based on current polling, and then using these support levels to estimate the parties’ chances of winning each seat in the Parliament. In this first post, we introduce the poll-based element of this model; in later posts we will introduce and explain the seat-based element.

The past is, of course, an imperfect guide, as voters and parties change and each election is, to some extent, unique. However, this does not mean past polling tells us nothing. On the contrary, as we have shown in previous research (non-gated version here), careful analysis of past polling reveals common underlying trends and patterns in British public opinion. It is these trends and patterns which we use to estimate our forecast of the likely path of public opinion. In 2010, our method fared relatively well against the alternatives on offer, getting the overall outcome of a hung Parliament with the Conservatives as the largest party correct, and coming quite close on all three parties’ seat totals. The forecast performed well relative to others forecasting models published by colleagues, notably beating celebrated polling oracle Nate Silver’s prediction for the British election.

The method works in the following way. Thanks to the Fixed Term Parliaments Act, we know how many days remain until the next general election. For any given day, we can use all the polling data from past general election campaigns to estimate two things: how closely the current polling is likely to reflect the election outcome, and which direction public opinion is likely to move in between now and election day. We do this for each of the three main parties separately, seeing what polling history can tell us about their respective fates.

This is one of the simplest possible ways of forecasting how elections will turn out, and it leaves out an awful lot. We do not look at the impact of leader approval ratings, the objective state of the economy, or public economic perceptions – things which other models have used as forecasting tools. We simply take the best possible estimate of where public opinion is today (an estimate constructed using our poll aggregation method) and ask: How informative does history suggest this estimate will be as a prediction of the next election? Where does history suggest public opinion will move between now and election day?

Our method starts by considering the systematic and predictable ways in which the public’s intention to vote for parties varies over the election cycle – based on past evidence. Some shifts in public opinion are impossible to anticipate, such as in reaction to shocks or events. Other dynamics may be more predictable, however; for instance that pre-election poll leads tend to fade or that parties may benefit from ‘campaign effects’ (such as due to increased attention during the official election campaign). To forecast the election day vote share, we need to know the relationship between vote intention t days before the election and the vote share in past elections. Therefore, the first step in our forecasting procedure is to estimate the relationship between vote intention and vote share through a series of regressions – for each of the main parties – for each day of the election cycle. To do this, we use all available polling data since 1945 (more than 3,000 polls) – across seventeen elections. This allows us to determine both how well the polls predict the final outcome on a given day (unsurprisingly the polls become more predictive the closer we get to the election), and to determine whether support for a party is above or below its long-term equilibrium level – and is likely to gain or fade as the election approaches.

Our past work, using this very simple method, suggests it has some useful lessons to teach us. Firstly, we find that the predictive power of polls evolves differently for different parties: polling of Conservative support becomes steadily more predictive from over a year out, while for Labour and the Liberal Democrats, the main improvement in accuracy comes in the last six months. Secondly, we find that support for Labour, the Liberal Democrats and the Conservatives tends to “regress to the mean” – if current support is above the long run average, it tends to fall; if it is below, then it tends to rise.

While the daily regressions teach us a lot, there is a fair bit of “noise” in the regression estimates, as each regression is based on 17 data points (elections). However, although each regression is based on only 17 data points, we have a separate regression for each day and we know that the estimates for one day should only differ so much from the estimates from the days immediately before and after it. Therefore, the second step in our forecast procedure is to reduce the noise by smoothing the regression estimates over time. The procedure we use to produce the smoothed regression estimates is similar to the procedure a sound engineer would use to remove static from a sound recording.

The third step in our forecast is to use the smoothed regressions estimates to produce forecasts by plugging our daily estimates of vote intention into the smoothed regression equations. These vote intention estimates come from the same poll aggregation method we use in our monthly Polling Observatory updates – a Bayesian averaging of the polls.

In the final step, we pooled the forecasts over 30 day intervals, so that the new pooled forecast on each day is a Bayesian averaging of all forecasts up to 30 days prior to that day.

The predictions we get by applying these methods to current polling since March 2011 are shown in the figure below. Our forecast model has consistently predicted a very close result – the Conservative vote share is expected to recover from its current level of around 32%, rising to around 36%, within half a percentage point of Labour, whose poll share is not expected to change much from current levels. In vote share, the result is close to a dead heat – the Conservatives are currently forecast to have 36.1%, and Labour 36.5%. The Liberal Democrats are forecast to recover some ground from their current polling position, but still put in their weakest performance in decades, with a forecast vote share of 10.1%.

Forecast 01-05-14 cropped

Forecasting vote shares can only take us so far, however. Westminster elections are won and lost in 650 separate battles for constituencies up and down the country. The aggregate vote shares are only an imperfect guide to the likely distribution of seats – our current forecast of a 0.8 point Conservative lead in vote share, for example, would be likely to produce a Labour seat lead. In future posts, we will employ the second part of our forecasting model to translate these vote shares into seat shares.

The seat-based section of the forecast also provides us with a mechanism to examine two of the big unknowns in our forecast – how the Liberal Democrats will perform after their first term in government for generations, and the impact of UKIP. Neither has any historical precedent, so we cannot model these effects in the historical part of the model. However, we can take an alternative approach, taking the baseline predictions from our historical model and applying different scenarios in the seat based part of the model. This will give us some sense of how sensitive the likely outcome is to changes in the fortunes of the two smaller parties. We will explore such scenarios in future posts.

Robert FordWill JenningsMark Pickup and Christopher Wlezien.

Beyond the Youth Citizenship Commission- Energising the Debate on Youth Political Participation

By Dr. Andy Mycock, University of Huddersfield

As part of its on-going commitment to promoting political activism and democratic engagement, researchers representing the Centre for Citizenship, Globalization and Governance (C2G2) have played a leading role in the development of a Political Studies Association (PSA) project seeking to enhance youth citizenship. Gerry Stoker, Professor of Governance and Director of C2G2, and postgraduate research student, Emily Rainsford, have contributed chapters to a new volume, Beyond the Youth Citizenship Commission: Young People and Politics. The publication is edited by Professor Jon Tonge (University of Liverpool) and Dr Andy Mycock (University of Huddersfield), who between 2008 and 2009 served on the independent Youth Citizenship Commission (YCC) formed by the last Labour government. The volume builds on the work of the YCC and applied research by members of the Political Studies Association and seeks to further energise debates about young people and democratic participation. The C2G2 played an important role in developing the project and publication of the volume, kindly co-sponsoring the hosting of a workshop for contributors in London in January 2014.

Contributions to the volume provide short opinion pieces on a range of youth citizenship topics and offer policy proposals to encourage governments, political parties and youth stakeholder groups across the UK to adopt more dynamic approaches to encouraging young people to get involved in politics. The volume addresses issues such as votes at 16, political participation of young women and BME groups, citizenship education in schools and universities, youth social media, and compulsory youth voting.  Policy proposals include a referendum on votes at 16, compulsory electoral registration in schools, and a call for the Westminster All-Party Parliamentary Group for Women in Parliament to establish an inquiry to encourage more young women to participate in politics.

Gerry Stoker’s contribution to the volume explores political citizenship and the innocence of youth. He argues that rather than despairing about the relative non-engagement of young citizens in formal politics, there is a need to recognise the positive aspects of their relative divorce from politics and their relative lack of cynicism and fatalism. Gerry suggests that young people are more open to the prospects for change and doing politics differently. A different political order could, he suggests, be stimulated by reforms such as lowering the voting age to 16 and young people’s national representative parliaments, assemblies and forums across the UK being given the right to call annual ballots or referenda on topics of their choosing.

Emily Rainsford draws on her doctoral research to identify a number of reasons why political parties are struggling to recruit young people to their ranks. She suggests that the relationship between young people and political parties is complicated but there is an urgent need to address the causes of party political disengagement. The need for new approaches to youth political party membership requires an acknowledgement that young people are adopting distinctive forms of political activism. The terms of party membership should therefore be reviewed to increase opportunities for young people to be able to influence the design of policy and develop forms of participation that reflect their interests.

The volume was discussed at a special panel at the Political Studies Association conference in Manchester on April 16th 2014. A panel of respected academics, including Gerry Stoker and Jon Tonge, discussed youth citizenship issues with representatives from the Manchester Youth Council and local schools. The panel also included Sam Johnson, a young councillor from Manchester City Council and Ian Wybron, a member of the Demos Generation Citizen project.

The volume will be officially launched at an event in Westminster in the summer and other events will be hosted in Edinburgh, Belfast and Cardiff. Chapters from the publication will also be hosted in an exciting series of on-line debates on the Democratic Audit and PSA blogs, with leading politicians, academics, and youth organisation will respond the policy proposals.

An electronic copy of the Beyond the Youth Citizenship Commission: Young People and Politics volume can be downloaded here.

For further details of PSA youth citizenship events linked to the project or to ask for a hard copy of the volume, please contact Dr Andy Mycock.

Polling Observatory #36: Farage’s Spring Uprising

This is the thirty-sixth in a series of posts that report on the state of the parties as measured by opinion polls. By pooling together all the available polling evidence we can reduce the impact of the random variation each individual survey inevitably produces. Most of the short term advances and setbacks in party polling fortunes are nothing more than random noise; the underlying trends – in which we are interested and which best assess the parties’ standings – are relatively stable and little influenced by day-to-day events. If there can ever be a definitive assessment of the parties’ standings, this is it. Further details of the method we use to build our estimates of public opinion can be found here.

UK 04-30-14 cropped

This month’s Polling Observatory suggests that, not for the first time, Nigel Farage’s UKIP uprising has thrown a spanner into the Westminster political machine. The slow but steady tightening in the gap between the big two parties comes to a halt this month, as both lose ground while UKIP surges. Our estimate for Labour this month is 35.3%, down 0.9% on last month, and continuing the past year’s pattern of slow but steady decline. Labour have now hit their lowest point in the polls since Ed Miliband was elected leader of the party in the autumn of 2010. However, the Conservatives have not been able to capitalise on Labour’s continued decline as their support has fallen even more sharply this month, down 1.4 points at 31.6%. The Liberal Democrats have also seen no electoral benefit from their leader’s high profile combat with Nigel Farage over the EU – we have them down 0.2% this month at 7.4%.

All the momentum, and the media focus, lies instead with Nigel Farage and UKIP, whose surge in European Parliament polling is, as we predicted in earlier posts, being echoed in Westminster polling: UKIP stand at 14.1% this month, close to their highest ever share of 14.4% achieved in the aftermath of their local election success last summer. With UKIP currently favourites to win the European Parliament election, and become the first new party in nearly a century to top the poll in a nationwide election, further advances in the next few months look likely. It is likely that the upcoming Newark by-election will take place with UKIP support in domestic polling at record levels.

The current UKIP surge, however, does owe something to the unusual political context with a European Parliament election looming. In each of the past two European elections, UKIP has advanced in the polls, as their defining issue dominates the political agenda, and they expand beyond their traditional base of disaffected, struggling working class voters angry about immigration by winning over better off, often Conservative-leaning Eurosceptic voters who find UKIP attractive in European elections but less so in Westminster ones. In past election cycles, UKIP has struggled to hold on to these “strategic defectors” – voters who view UKIP as a vehicle for sending a message about the EU, but not a natural political home.

It remains to be seen whether the same dynamic will assert itself in the rather different political climate of 2014 – UKIP’s much stronger poll standing, greater media coverage, and improved financial and organisational resources may make it easier for them to retain new recruits. However, the Conservatives will certainly be hoping that the pattern of 2009, and (to a lesser extent) 2013 repeats itself – with UKIP’s spring surge fading away as voters return to the domestic political agenda and start to drift back to the mainstream parties, boosting Tory vote shares in particular.

Labour will hope that, with UKIP now targeting both main parties for votes, the decline in their support this month also reflects the temporary inflation in UKIP support, and that they too will recover more ground if and when UKIP decline than was true before. This is certainly possible, but it might be unwise to pin their hopes on a Farage fallback. The fall in Labour share this month was the continuation of a now well established trend that has seen the party shed 7 percentage points of support since their peak in the summer of 2012. Those within the party pushing for a “35% strategy” might be a little concerned that Labour support has already fallen back to this level, with over a year to go, and clear evidence of a downward trend in support. It is quite plausible that the remaining Labour backers are more firmly behind the party than those who have leaked away over the past 18 months, but the continued loss of support, and the precarious lead remaining, must give Labour plenty to worry about.

Those within all the parties wondering what today’s polling means for next year’s election will have some additional information to chew on next week when, exactly one year ahead of the general election, we launch the Polling Observatory forecast model, which utilises historical polling trends to estimate the likeliest pattern of results next May based on where things stand today. Tune in on May 7th to find out our current estimate of where support for the parties will be come election day.

Robert FordWill Jennings and Mark Pickup

A Debate on Ukraine, Crimea, Russia and International Law’s (In)Capacity

Tonight, academics from Southampton Law School and Southampton University’s Department of Politics and International Relations will debate the place of international law in the Crimean crisis and ask what it means to brand Russia’s action ‘illegal’. The debate will be held tonight at 6pm in Building 46, Lecture Theatre A (Room 3.001).