The Shotgun Blog
Wednesday, April 30, 2008
Deconstructing polling: Asking the right questions and pricing opinions
Web- and email-based polling has allowed companies like Angus Reid Strategies to pump out timely surveys on current affairs. It’s cheap, fast and likely as accurate as traditional telephone polling, so we can expect to be flooded by survey data on issues like polygamy and the “in-and-out” controversy.
This is great. Polling data is valuable to policy makers and opinion leaders, and interesting to the media and anyone who pays attention to social, political and economic trends.
But readers should always treat polling data with scepticism. (Western Standard readers likely already approach polls with scepticism.) Below are the results of a traditional phone interview poll conducted by pollster Dr. Faron Ellis some time ago. I was an executive with the company at the time and wrote the questions to test a theory: You can get radically different results when the same issue is probed using radically different, but equally fair, language.
Here’s my experiment, never before released, in the form of four survey questions on economic protectionism. In general, I wanted to learn if Canadians were protectionist. In particular, I wanted to know what Canadians might think about anti-dumping duties on Chinese bicycles, which was news at the time.
1. Should Canadian consumers pay more for goods and services in order to protect the profits of Canadian manufacturers?
Less than 20% of respondents support protectionism when consumers would be expected to pay more to protect the profits of Canadian manufacturers. That’s good news for free traders.
2. Should Canadian consumers pay more for goods and services in order to protect Canadian manufacturers against foreign competition?
When you make reference to “foreign competition,” the number of respondents who support protectionism moves from 19% to 28%. (You can call this additional 10% a nationalist or xenophobic demographic depending on your perspective.) But support for protectionism is still under 30%.
3. Should the government of Canada protect Canadian businesses against foreign competition?
When you shift the burden of protectionism from consumers to the government, you get a strong protectionist response. Respondents are saying “I don’t want to pay more to protect Canadian businesses, but the government should.” Of course, when the government takes on this burden, consumers pay a hidden cost through their taxes and higher cost goods and services. There’s a "Public Choice" economic lesson here:
The costs of such inefficient policy are dispersed over all citizens, and therefore unnoticeable to each individual. On the other hand, the benefits are shared by a small special-interest group with a strong incentive to perpetuate the policy by further lobbying. The vast majority of voters will be unaware of the effort due to rational ignorance. Therefore, theorists expect that numerous special interests will be able to successfully lobby for various inefficient policies. In public choice theory, such scenarios of inefficient government policies are referred to as government failure — a term akin to market failure from earlier theoretical welfare economics.
4. Should the government of Canada protect Canadian jobs against foreign competition?
When the question is changed from protecting companies to protecting jobs, you see a 10% increase in the protectionist sentiment, bringing it to almost 70%. How do you protect jobs without protecting the companies that create those jobs?
While each question in this poll measures attitudes toward protectionism, the responses range from strongly anti-protectionist to strongly pro-protectionist depending on the wording of each question.
So what’s the conclusion of my experiment?
First, polls may be an accurate measure of the questions asked, but how the questions are worded has a huge impact on results. (This also reminds us that the words we chose have a huge impact on the effectiveness of our communication. If you want to advocate against protectionism, you’ll have more success reminding consumers that they will ultimately be paying more to protect corporate profits.) On this point, the Penn & Teller: Bullshit! TV show on Showtime took at look at how Republican pollster Frank Luntz “manipulates” polls. (In my mind, Luntz understands polling better than almost anyone in the business. He sees polls primarily as a way to test the impact of political language.)
Second, opinion polling is often meaningless as respondents will support virtually any government “feel good” initiative as long as these initiatives are seen as having no direct costs. The costs, of course, are hidden and widely disperse in the form of taxes and/or slightly higher costs associated with uncompetitive markets. When respondents are asked to pay directly for their positions, they give more honest answers.
So the next time you read a poll, ask yourself if a differently worded question would get an opposite result. Then ask yourself if respondents are being asked to directly internalize the cost of whatever it is they are supporting.
Or you can take Penn & Teller’s advice and just yell “Fuck you, Frank!” every time you see a poll. (Those boys are so zany.)
You can read the complete results of this experimental poll here: Download jmck_polling_economic_protectionism_august_2003.pdf
Posted by Matthew Johnston on April 30, 2008 | Permalink
TrackBack URL for this entry:
Listed below are links to weblogs that reference Deconstructing polling: Asking the right questions and pricing opinions:
Question: Do you think Matthew Johnston's posting on polling was informative and useful? YES!
Phrased differently: Do you think a posting by ultra-libertarian blogger Matthew Johnston, on how to manipulate poll questons, gives us an insight into the dark, Machiavellian machinations of the vast right-wing conspiracy? HMMMM, NOW THAT YOU MENTION IT.....
Just kidding, Matt. Good item!
Posted by: Terry O'Neill | 2008-04-30 11:00:15 AM
Excellent. For those who do favor protectionism, I propose they willing pay more for the goods they buy than what would otherwise be a free market price. I would not want to deny them of the warm fuzzies they would get by helping others. And I would not be forced by law to pay more than the market price.
Posted by: TM | 2008-04-30 11:09:16 AM
>"(In my mind, Luntz understands polling better than almost anyone in the business. He sees polls primarily as a way to test the impact of political language.)"
Test the impact of political language is a euphemism for "designing better push polls to influence the direction of political discourse and perception".
Posted by: Speller | 2008-04-30 12:14:38 PM
Poll sample sizes are to small to influence the direction of public opinion via the process of surveying. But knowing what phraseology works to elicit the "right" response, is valuable. Like it or not, and I don't, that's all politics is really about. It's also how the Republicans won the 1994 “Contract with America” campaign, according to Luntz.
Posted by: Matthew Johnston | 2008-04-30 2:04:48 PM
Very good article, Mr. Johnston.
FUCK YOU, FRANK!
Posted by: Marc | 2008-04-30 4:11:05 PM
and of course, my favourite example, courtesy of Yes Minister!
Posted by: sabre0 | 2008-05-01 10:58:21 AM
The comments to this entry are closed.