Monday, December 13, 2010

How you should look at polls

Anthony Wells (no relation to Paul Wells), author of UK Polling Report, an excellent website that gives you daily updates about polls and politics across the pond, had these sage and wise words for anyone wanting to take anything away from polls:
You should resist the natural tendency to believe the polls we want to and question those that don't. The most useful polls are often the ones that bring you bad news, so use them to address weaknesses in strategy, rather than try to explain them away.

When different polls show conflicting results, try to understand why. Human beings can't really be put into neat little boxes labelled 'support' and 'oppose', yet that's what most newspaper opinion polls do. Different question wording and different approaches to an issue will often paint a different picture and the best way to understand what the public think about an issue is to look at all the polls, not cherry-pick those that tell you what you want to hear.
These words are too intelligent by any measure, yet the fact that they are so plainly obvious, and still a good portion of pundits, whatever their stripes,  refuse to follow that advice, amazes me.

When people start to treat polls as the most exact thing possible, and does it while ignoring polls going against their narrative, they start to suffer. A poll by itself is not accurate; you need to look at singular polls in context of what the entire polling universe is saying at the same time. If it is truly out there, showing results different from what every single other pollster is saying, and whether or not it proves to show a beneficial result in your favour, then you should realize that maybe you should take a closer look at the poll, compare it to others, whether it follows a trend, bucks it, or takes bits and pieces and presents its own credible result, and otherwise critically analyze it. When you refuse to do this, not only are you basing your reactions and decisions on wrong information, but you're profligating that information to everyone else. It may help reinforce a narrative you want, sure, but reality is not going to bend to your will so easily; it'll come back, and with a vengeance. And once you've involved all your ideas, strategies, and reactions into this false narrative, there's no going back. You, your plans, your reputation, and everything else will suffer, while the other guy gains a leg up. Not something you want.

So, for all pundits, take a look at Mr. Wells' article, and take heed of the advice it gives. It'll probably save your ass more than once in our brave new partisan world.

2 comments:

  1. You start with an expert's level-headed advice not to cherry pick polls, but then you go on to extract an entirely different conclusion (that if one company's is out of line with the rest, that it's most likely wrong). Sorry, but that's not the appropriate concl. to draw, there (after all, the odd one out might have a much larger sample, and far less leading or biased q's (incl. the order of q's), or a better sampling frame or methodology [phone vs net, e.g.]).

    A better concl. would be to advise people to look at the trendline within each co's series (assuming they cleave to the same q's & methodology from survey to survey) & compare those, to see whether there seems to be any genuine movement.

    ReplyDelete
  2. If you drew different conclusions to what I posted, then I apologize, but I did intend to say what you think is the proper conclusion. Maybe I should have been more clear, and I'll edit it to reflect that.

    Thank you.

    ReplyDelete