Friday, October 13, 2017

How to read the competing Calgary election polls

With just two days left before most Calgarians finally get their election over with, dueling poll releases from Mainstreet Research and everyone's favourite Forum have painted some very different pictures of the race.

Let's start with the Mainstreet poll, which gives challenger Bill Smith a healthy 13-point lead over incumbent Naheed Nenshi.

The September 30th Mainstreet release for Calgary caught some flak from myself and more respectable quarters over the odd crosstabs that gave Nenshi a lead among men but a severe deficit among women, a breakdown we wouldn't usually see given the left/right divide between Nenshi/Smith. It wasn't impossible, but it seemed odd. That red flag doesn't pop up here, though there is a potential brow raise with the 18-34 number - again, not impossible, but not what you'd expect.

However, Mainstreet went more in-depth, asking favourable respondents of the two main candidates what they believed their best qualities were, and also gathering general approval/disapproval. Nenshi lagged behind Smith regarding taxation and has a 48% disapproval rating, which can go a long way as to why the mayor is running behind his opponent.

Now on to Forum's release, conducted on behalf of the Canadian Municipal Election Study:

Forum gives Nenshi a substantial 17-point lead in the race, but I would a caveat to it. Much of the lead comes from 18-34 demographic, which while I believe would be more likely to vote for Nenshi I'm not so sure they would go that heavily. There is also the broader question regarding turnout scenarios and the sample sizes presented - for example, 18-34s turnout isn't great versus 65+, yet there is a greater sample of the former than the latter in this release. (Mainstreet's samples, in comparison, were more equally sized, though you could latch on the same criticism, as they had twice the 18-34s as they did 65+, and once again Smith led among both substantially.)

However, if you were to ask me personally where I thought the race was in Calgary, Forum's numbers fit the narrative I've seen more closely. That is of course subject to all my biases and can't be trusted, but while I can easily see Smith gaining traction, common sense says that Nenshi is still popular and it would take a large scandal, and not just general fatigue over taxes, to turf him. Some could point to the arena issue if they wish, but Mainstreet's numbers won't back them up, as very low priority was given to that squabble by respondents.

But of course, we want a more objective measure than my gut. Following the initial controversial release my Mainstreet, a lot has been made over the various methodologies of the polls involved. So let's just get through this quickly.

Mainstreet conducts polls using Interactive Voice Response (IVR), which is to say you get a call from an automated voice asking you to press a key or say a word corresponding to a question asked and answer given. They included both landlines and cells.

Forum's poll was conducted online, through its respondents were recruited through responses to an automated dialing service.

Another poll done by a different company was done through an online-only panel that, while somewhat self-selective in my eyes, would still give a good cross-section of voters (it also showed a strong Nenshi lead).

All of these methods are completely valid and have similar track records.

IVR vs online vs interview all have good and bad moments and neither can truly claim to be better than the other, so all the huffing and puffing over this and that is meaningless. Any of these polls could be right, though of course only one will be.

There is no objective way to determine who is right until Monday night rolls around. But we can discuss why these numbers are so divergent.

The main problem is that municipal elections are very hard to pin down, even in cities the size of Calgary. Turnout is often low, lack of partisan affiliation can lead to odd alignments among voters, and the reasons behind a voter's decision can be obscure and hyper-local (for example, how will the raised issue of water flouridation in Calgary change turnout among voters who consider that a priority?). While the size of Calgary means we'll see more general swings on more standard issues, it's very hard to properly reflect what drives a lot of people to the polls in these elections.

With the above taken into account, getting a random sample of opinion from 1,000 to 1,500 people about this race will be trickier than usual. Consider as well that being forced to choose from a smaller pool of potential voters (just those in Calgary vs all of Canada) means your margin for error is larger, and you can have a pretty chaotic and fluid situation. Getting a result within the margin of error in either case will would surprise me, and we could easily see a close two-way battle rather than these large distances between the candidates, or Chabotmania. We just don't know and are making educated guesses at best here.

To end on a cliche, the only poll that really matters will be election day, so take everything else with a grain of salt - good advice for any polling you come across.

No comments:

Post a Comment