« Up in arms | Main | "America, this is what a feminist looks like" »

The Sordid Business of Opinion Polls

As I was riding the Park n Ride to work one morning last week, the bus passed a fortune-teller's store, which had gone out of business. The place had been doing poorly for a very long time. As I pondered the painted word 'psychic' on the side of the building, I wondered why the medium did not know this was going to happen. For all of that, there are many fortune-tellers who make good money, telling folks stories which sound good, which is what the customers really want. Of course, there are many people who won't be taken in by horoscopes and fortune-tellers; they know better than that. Yet, many of those fine people read the poll reports in the papers and watch the polls in television, so they will know who's going to win the election. They believe the claims of people they do not know about what will happen in the future, just because they like the story.

I have followed opinion polls for some years now, and I have great respect for a poll which is properly done, and properly reported. Unfortunately, there are a lot of polls which get published without providing readers or viewer with the proper context, and worse there are a number of major polls which hide vital parts of their internal data. The media, it must be remembered, is not and never has been in the business of reporting facts. Instead, they want to sell a story, in order to claim lots of attention which will increase readership or ratings. And in this election, the media has overwhelmingly been biased in favor of Obama.

Before I continue, I need to go back and say again how a poll should be properly done. There are two major authoritative poll accreditation bodies;

The National Council on Public Polling (NCPP), and the American Association for Public Opinion Research(AAPOR). Any valid poll should include a disclosure that it complies with NCPP or AAPOR standards of methodology, but most do not do so.

The NCPP has published a guide for journalists to use when discussing the report from an opinion poll. Some of the questions they say a journalist should ask include

Who did the poll?
Who paid for the poll and why was it done?
How many people were interviewed for the poll
? (This, by the way, is why the state polls are less statistically valid than the nation polls. They usually involve a much smaller poll of respondents than the national polls use)
How were the people chosen to take part in the poll?
What area were those people from?
What was the response rate to contact attempts?
(almost no one answers that one)
How does the poll describe the results?
What questions were asked, and how were they worded?
What order was used for questions?
(this is an old, old trick - if a controversial issue is asked just before a certain candidate is named, often this creates a false connection between the candidate and the controversy in the mind of respondents)
What events are likely to have impacted the poll results?

The plain fact is, that almost no media outlet or polling group reveals the answers to all of these kinds of questions. And that damages their credibility as valid reflections of the public's opinion.

The AAPOR warns that polls almost always fall into one of seven categories of sponsorship:

1. Academic institutions
2. Federal, State and Local Governments
3. Media Organizations
4. Non-profit groups or Foundations
5. Special interest groups
6. Businesses and Corporations
7. Political campaigns, consultants, and candidates

They also warn about many of the same problems in poll reporting as the NCPP, and the AAPOR goes further to warn that many of these sponsors may be biased, even when they claim to be objective. The first question, in any poll result, is 'who benefits'?

This brings me to Rasmussen. In an earlier article, I rather harshly said that Rasmussen was in it for the money. I have always asked the questions recommended by the NPCC/AAPOR, and in doing so I have noticed how some major polls refuse to answer important questions. I learned a long time ago that if you had enough information, you could work out the mechanics of most major conclusions. I found Gallup, AP-Ipsos, and CBS News very open about their methodology and the internal data of their polls. But I have always been disappointed in the secretive and sometimes disingenuous behavior of Zogby, NBC, ABC, and Rasmussen. As it happens, Scott Rasmussen has enjoyed a lot of success at his business, but if you want internals it will cost you, and if you want to reverse-engineer his polls to test for bias, he simply won't allow you a way to have all the information needed. That, again being blunt, strikes me as both mercenary and essentially dishonest. And therefore, no matter what his polls say I will always see a little red flag next to his published claims. My rule of thumb is very simple - it does not matter whether the poll says what I'd like to hear, it matters whether all the supporting data is available.

Another thing that is troubling, is the desire for instant gratification. For example, Sarah Palin clearly beat Joe Biden in the debate they had last week (although ardent democrats are just as sure Biden triumphed), and so there are people on both sides looking at the current polls for signs of their guys big win. But there are two reasons why you will not see that, at least right now. First, even the invalid daily tracking polls take their results from three days of polling, which means the first poll with full post-debate reaction will not be released until Monday. And I have to warn you that the polls will not reflect an immediate valid effect - if they show a sudden jump for either side it is far more likely to mean that the polls have - once again - changed the party affiliation weights, a practice as dishonest as putting one foot on the floor while you weigh yourself. The reason is because while people do change their minds, historically it has always been a few people at a time and in small steps. One proof of that is that as popular as Presidents Reagan and Clinton were, neither saw a major shift in party identification during his Administration. People supported the man, not necessarily the party, and in this election it would foolish indeed to imagine that people who supported Biden were also becoming more pro-democrat than before, or that those who supported Palin were becoming more pro-republican than before. After all, the biggest statistical group of undecideds consider themselves to be independents, and while they will come over to one candidate or the other, that does not lock them in with the whole party. Party identification should be a static measure for the duration of a campaign, and manipulating party affiliation is simply cheating the results to create a story out of imagination.


TrackBack URL for this entry:

Comments (7)

Excellent article DJ !!!!!!... (Below threshold)

Excellent article DJ !!!!!!!!!!!!!!!!!!!!!!!!!!

Good show DJ.<... (Below threshold)

Good show DJ.

Just to add justification to what you said, ALL politics are local. This is especially true with party identification. If most of your neighbors are Democratic, then you will tend to be one too. As individuals shift their political allegiance, they do so slowly because chances are, one or more of their friends is still hanging on and griping about how his/her party is screwing up.

It took a heckofa long time for the South to shift from Democratic to Republican and the Northeast from Republican to Democrat. I bet most folks understand how slow party affiliation changes if they think about it.

Honestly I don't think the ... (Below threshold)

Honestly I don't think the VP debate, or even the presidential debate for that matter had much of an impact on poll numbers. There is always this assumption that it will, or had a large impact but this is not necessarily the case and historically debates have had a very small impact unless the candidate just hits it out of the park.

Also, as I understand it there is a rather academic debate about whether weighting for party ID is a good idea.

Thanks D.J. You know I kno... (Below threshold)

Thanks D.J. You know I know the media can use the polls to make news and to dishearten voters. I don't want to believe them; I didn't until the 2006 elections.

But whatever they say, I will vote and I just put a check in the mail, and I'm gonna start calling for the campaign.

"[T]he media has overwhelmi... (Below threshold)

"[T]he media has overwhelmingly been biased in favor of Obama."

The operating word here is "orgy", D.J....

BTW: Great article!

Even your idol Karl Rove, u... (Below threshold)

Even your idol Karl Rove, using his own methodogies has Obama far ahead of Palin/McCain.


You guys are still in the denial stage.

Rove's "methodology" is jus... (Below threshold)

Rove's "methodology" is just stealing a bunch of polls and averaging them. Which leads right back to the question of the methodology of the polls he's averaging.






Follow Wizbang

Follow Wizbang on FacebookFollow Wizbang on TwitterSubscribe to Wizbang feedWizbang Mobile


Send e-mail tips to us:

[email protected]

Fresh Links


Section Editor: Maggie Whitton

Editors: Jay Tea, Lorie Byrd, Kim Priestap, DJ Drummond, Michael Laprarie, Baron Von Ottomatic, Shawn Mallow, Rick, Dan Karipides, Michael Avitablile, Charlie Quidnunc, Steve Schippert

Emeritus: Paul, Mary Katherine Ham, Jim Addison, Alexander K. McClure, Cassy Fiano, Bill Jempty, John Stansbury, Rob Port

In Memorium: HughS

All original content copyright © 2003-2010 by Wizbang®, LLC. All rights reserved. Wizbang® is a registered service mark.

Powered by Movable Type Pro 4.361

Hosting by ServInt

Ratings on this site are powered by the Ajax Ratings Pro plugin for Movable Type.

Search on this site is powered by the FastSearch plugin for Movable Type.

Blogrolls on this site are powered by the MT-Blogroll.

Temporary site design is based on Cutline and Cutline for MT. Graphics by Apothegm Designs.

Author Login

Terms Of Service

DCMA Compliance Notice

Privacy Policy