Tuesday, January 24, 2012

Climenhaga's Wrong on Forum Polls

This. This article by David Climenhaga on the recent Forum poll, who is an otherwise excellent writer out of Alberta, is another reason why I can't read Rabble.ca for any length of time without having a meltdown.

The entire premise by Climenhaga is that Forum's two recent polls - one in December and one out this month - are showing inconsistent results compared to the rest of Alberta's many pollsters, in mainly two ways: higher Wildrose support, and lower PC support, the latter being a special concern due to the fact that Redford should have had a honeymoon period which didn't show up in Forum's polling.

Let me dispel both of these misunderstandings right now.

Issue #1: Wildrose support too high

No. It isn't. Except for the January poll, Forum's results for the Wildrosers remain consistent with other pollsters when you allow for a margin of error and the fact that there will be about a 5% variance between pollsters. It's easy to see.

I think the fair spot to take from would be the polling done since September 2011, which is when the field for the PC leadership was fleshed out (Redford became leader on October 2nd). That gives us six polls to work with, which are the following:


As you can see, the only outlier that Forum shows is in the most recent poll. Their poll in December fit in snugly with the other four polls estimated Wildrose support. The problem Forum has now is whether or not that 29% they show this time will be confirmed by other pollsters. And frankly, 29% to my personally seems like a reasonable level of support.
So this sarcastic comment by Climenhaga - "Forum got it right and was simply the first to pick up on a dramatic developing trend of growing support for the Wildrose Party that appeared in December..." - is based on a misconception, since Forum did not show any "dramatically developing trend" of WRP support in December.

Only in January has WRP support peaked far above the other pollsters, and it's too early to say whether its an outlier or growing trend. However, it is dishonest to question Forum's result on the basis of supposedly outlier results in their December poll, which is simply not true.

Issue #2: PC Support is too low, other pollsters have it higher

In this respect, Climenhaga is indirectly right - 38% is on the low end of the scale for PC results since September, which usually have the PCs above 40%. However, it also isn't necessarily an outlier result; given the margin of error (MOE) in both polls, the PCs could truly be above 40%, just on the low end of the scale. So this argument, while slightly valid, is still not entirely sound.

But that's not exactly what Climenhaga said. This is what he said - "There's something wrong with Forum's sample that is producing wildly different results from those of most other polling companies, which have put support for Redford and the Conservatives at around 50 per cent or better." - which, unless Climenhaga has access to polls I don't (entirely possible, but unlikely) is just not true.

There's two ways to tackle this. One, there's just the raw numbers - only one pollster shows the PCs above 50% (Environics), and a second comes close with 48% and with the MOE could reach 50% or more (Lethbridge). The other two pollsters are below 45%, and Forum is below 40%. While saying "half of the pollsters show the PCs high" is a valid argument, we come to the second way to tackle it.

There's only six polls to chose from, and they're spread across six months! While it's one thing to say that pollsters show different results from other pollsters, the fact is that six polls - four polls, really, minus Forum's - spread across month-long periods is not a healthy base to compare to. There simply isn't enough up-to-date data to formulate specific trends on. You can say that Forum shows lower PC results than the other pollsters, which is true, but you have to admit two things while you say it: that some of the compared pollsters are old, and there's only four of them. Most of all, Forum is the only pollster to come out in December and January! There is no comparison to make because the other pollsters are from November and earlier. You can only speculate about a general trend, which Forum's results are perfectly normal to accept.


I do apologize for this wall of text but Climenhaga's article is just full of misconceptions and it annoys me. It's one thing to claim a pollster is showing outlier results or is a voodoo poll - I've said it a few times. But I do it on the basis of up-to-date results from other pollsters, and real proof that the numbers are truly outliers, much different from what any other pollster has shown. Most of all, I give pollsters the benefit of the doubt at first, because any poll that shows different results could be the start of a new trend.

Climenhaga, despite some of the cavaets he puts in his article, already seems set to conclude that Forum's polls are wrong. And here's this gem right here that shows you where some of the bias comes from:
If that poll's results return to the pattern of most voter intention surveys before Forum entered the field late last year, it will be strong evidence that the Conservatives are on their way to another huge majority, the NDP is the most likely party to form the opposition and that something is indeed wrong with Forum's sample.
As I have shown, Forum's polls are in line with the voter intention surveys of late last year - therefore you can't just up and conclude that another pollster which shows results similar to the other four pollsters contradicts Forum's polls. That is madness.

And what was that last part? Something about the NDP forming the Opposition? What? Only one poll - the Lethbridge College poll of October - gives the NDP enough seats to become the Official Opposition (7 compared to 3 for the WRP and 0 for the Liberals). No other poll ends up putting them in that position. And it seems, frankly, that the NDP are trending downwards anyways, whether you include Forum's polls or not.

Furthermore, Forum's results would be in line with the polling results before Stelmach resigned. That could easily point towards Redford losing her shine, though we'll see what happens. But that never gets mentioned, does it?

So that's it. I've shown and explained reasons why Climenhaga's article is a false premise and full of misconceptions. Ipso facto, I say.

5 comments:

  1. There were nine public polls of Alberta voters’ intentions in 2011, and one so far this year. There are in fact a couple of others that have been done for private groups and whose results have not been published, one of which I have seen and which affects my calculations.

    In the public polls, Progressive Conservative support ranges from a low of 33 per cent to a high of 54 per cent. Wildrose support ranges from a low of 16.1 per cent to a high of 29 per cent.

    Of the six polls cited in this author’s post, four use methodology that is questionable. Both Forum polls are based on automated calls and were conducted over a single evening. The Angus Reid and ThinkHQ polls use a self-selecting online panel. This does not mean their results are wrong, only that caution needs to be exercised in weighing their results against other polls using more tried and true methodology. Responsible pollsters make three to five attempts to reach a randomly selected household before giving up. Did this happen with either of the Forum polls? Unlikely given their time frames.

    In 2008, online polls did not produce accurate results, further reason for skepticism.

    Only one 2011 poll shows the NDP above the Wildrose Party, but this has never been the basis of my argument that the NDP is more likely to form the Opposition if the Progressive Conservative vote remains strong. Rather, that is based on my analysis of how the vote is likely to split in ridings where the NDP is strong versus ridings where the Wildrose is strong.

    My analysis is that of an engaged layperson with an interest in politics and polling. Feel free to disbelieve it! Making bets based on my predictions is not recommended. But while I have had some spectacular failures (calling the Calgary Mayoral election for Barb Higgins, for example) my overall record ain’t bad.

    Differences greater than 6.2 per cent cannot be explained away by the margin of error.

    Finally, the caveats I put in my post are important. The “gem” you quote as evidence of my bias is given less weight in my article than the parallel point: “Forum got it right and was simply the first to pick up on a dramatic developing trend of growing support for the Wildrose Party that appeared in December.”

    I agree with this post that it is far from clear what is going to happen. I am betting, however, on another Conservative majority under Ms. Redford, with a side bet on the NDP becoming the Opposition.

    I won’t lose any sleep, however, if I am proved wrong.

    ReplyDelete
  2. This is not directed at the above, or, at anyone in particular, but rather at everyone, and, I will say it in allcaps.

    INDIVIDUAL POLLS ARE MEANINGLESS. THE ONLY THING THAT MATTERS IS POLLING AVERAGES FROM AT LEAST 5 DIFFERENT POLLS FROM AT LEAST 5 DIFFERENT COMPANIES. - AND IF YOU NEED A COMPARISON, YOU NEED AT LEAST 5 OTHER DIFFERENT POLLS FROM AT LEAST 5 OTHER DIFFERENT COMPANIES TO BE CERTAIN.

    I've tried to say that subtly for the past number of years, but a little allcaps cant hurt.

    ReplyDelete
  3. A note that the 5 companies from your second set CAN indeed be the same as the 5 companies from your first set. And if there are only 4 companies, or 3 etc doing polls for your province, then use what you got. PEI often only has 2 companies doing polls at election time, but the rule of golden 5's still holds, you really need at least 5 different polls to get a true average, even if you are forced to use polls from the same company. Also date is important, anything going back too far ruins it.

    Put in simple terms:
    One poll is meaningless
    Two polls from the same company are meaningless
    Two polls from different companies mean a LITTLE
    three polls from different companies mean more, etc etc etc

    ReplyDelete
  4. R by R overstates his or her case by suggesting that individual polls are meaningless, although it is true that the possibility of a rogue poll for any reason is greater.

    So it is obviously true that a number of polls with similar results tends to reinforce the conclusion reached by each pollster.

    In Alberta, however, we now have a situation where significant numbers of polls are breaking into one of two groups with markedly different conclusions.

    ReplyDelete
  5. David,

    I don't agree at all with your conclusion - I've seen no evidence of pollsters "breaking into two groups." If you'd provide these non-published polls and their numbers, maybe I could agree - unfortunately all I have is hearsay. I have what shows up as reported polling in the media and on the ever-useful Wikipedia. And based on that polling, there IS no split.

    I also disagree with your questioning Forum's and AR's methodologies (don't know enough about ThinkHQ to comment). Forum uses the same methodology used by other pollsters such as Nanos and Ekos (they may have moved away from it now, mind you), not to mention countless American and British pollsters, including Gallup, Ipsos, YouGov, and so on. AR's methodology, while I have a paranoia over as a general rule, is sound enough to produce results as accurate as any other pollsters. They were among the top, if not THE top, pollsters for both the 2008 and 2011 federal elections. You can question where their results skew a bit (I say it overestimates NDP support, something they do for the Lib Dems in the UK), but overall their methodology is sound.

    I also dispute your assumption of how NDP support will break, but that, unlike looking and determining the data we know, is speculative.

    My main you skipped over as well - the fact is that, as far as I know, Forum is the only pollster out this year for Alberta. As RxR pointed out, one poll does not only fail to qualify as a trend, but the fact is that you can't dismiss it either because there is nothing to compare it to! There simply isn't enough polling done by enough companies and in a short period of time to claim anything about Forum's polls.

    And I say this as someone who doesn't take Forum all that seriously sometimes, like in BC with that 34-23-23 result last month. Forum has a bad track record of skewing themselves sometimes, but unlike that poll in BC or the polls federally, there is no major outlier in Forum's polling, and until more polls are done which you can compare Forum to, that's simply the truth.

    ReplyDelete