« The Illness That Dare Not Speak Its Name | Main | Long time Detroit Democrat now supports McCain »

Poll Bias 2008: What the Numbers Say

Back in 2004, I determined a general measure of bias in the Presidential election by polls. I did the same thing for this year, which may be useful in examining the post-Convention stretch. I began by noting the 2-way poll results from Real Clear Politics from January 1 2008 through August 25.

I made a spreadsheet with the name of the group or agency which conducted the poll, the starting and ending dates of the polling, and the results for Obama and McCain. From this information, I drew up another spreadsheet which produced the daily average for the polls. I then compared each poll against the average for the days polled, to see where each poll stood against the average, noting whether Obama or McCain was getting more or less support than the average in a poll. I then was able to apply that information to determine the average bias, the aggregate variance, and essentially the stability of each poll.

I want to stop at this point, and make an observation about the fairness of these polls. As polls make statements which favor Obama or McCain, it is becoming common to see this poll or that dismissed as being unreliable or unbelievable. This is generally unwise, because in examining the polls presented, I was struck by how well, overall, most polls were conducted. While I have concerns about the demographics base used, and the weighting assumptions inherent in modern polling, the conclusions of each poll were consistent with other polls at the same time. Since twenty-six national agencies or groups have conducted national polls used for my research, such consistency indicates a high level of professional standards. However, within that acknowledgment, there were some clear trends and characteristics, which I think are worth mentioning.

As I said, there were twenty-six separate groups or agencies which conducted polls. Of these, six conducted only one or two polls this year (Battleground, McLaughlin, Franklin & Marshall, Ipsos,POS/GQR, and NPR). Accordingly, I have not considered these groups for the purposes of comparative bias.
That leaves another twenty groups, however, which have conducted a measurable pattern in their polls. For this report, I will note the five polls which most favor Obama, the five which are least favorable to Obama, the five which are most favorable to McCain, the five which are least favorable to McCain, and the five which are, as an aggregate, the least likely to be off the average measure for that date. I would like to emphasize that bias is used here not as an attack on the poll's integrity or veracity, but simply marks a tendency likely to continue in future polls between now and the election. It remains to be seen which of these polls will prove the most accurate in the final measure.
Here are the top five in each of the five categories, then:

Obama Favorable
1. ABC News/Washington Post (+3.33 pts)
2. CNN (+3.00 pts)
3. Quinnipiac (+1.97 pts)
4. Pew Research (+1.54 pts)
5. USA Today/Gallup (+0.92 pts)

Obama Unfavorable
1. Investor Business Daily/TP (-2.80 pts)
2. FOX News (-2.30 pts)
3. Cook/RT Strategy (-2.28 pts)
4. Hotline/FD (-1.62 pts)
5. LA Times/Bloomberg (1.26 pts)

McCain Favorable
1. Democracy Corps (+3.27 pts)
2. USA Today/Gallup (+2.83 pts)
3. Rasmussen (+1.89 pts)
4. CNN (+1.51 pts)
5. ABC News/Washington Post (+1.40 pts)

McCain Unfavorable
1. Investor Business Daily/TP (-4.55 pts)
2. CBS News (-2.80 pts)
3. Hotline/FD (-1.88)
4. Quinnipiac (-1.57 pts)
5. Time (-1.40 pts)

Aggregate Stability
1. NBC News/Wall Street Journal (1.81 avg total variance)
2. AP-Ipsos (2.06 avg total variance)
3. Gallup (2.17 avg total variance)
4. Pew Research (2.40 avg total variance)
5. CBS News/NY Times (2.53 avg total variance)

I will let the reader draw his/her own conclusions, after making three suggestions:

1. Notice which groups are favorable or unfavorable to both Obama and McCain

2. Aggregate Stability is a near-unknown quality, but represents a reason to have great confidence in a poll. If a poll does not miss the average by very much, it represents a reliable place for quick vitals.

3. Most of the variance shown is within the standard deviation for the population base used in the polling.


TrackBack URL for this entry:

Comments (7)

I'm sure there's a good mat... (Below threshold)

I'm sure there's a good mathmatical explaination, but could you tell us (in layman's terms) how USA Today / Gallup is biased favorably towards both Obama (#5) and McCain (#2)?

This is a good post,... (Below threshold)

This is a good post, DJ. I'm going to share it with our statisticians and see if it doesn't tell us anything about our own company (we conduct polls for both major parties and for general consumption in the U.S.) and/or our competitors.

Just curious, but what do you conclude about IBD/TP's negative skew towards both candidates? That they're pessimistic about either candidate's prospects of steering the economy out of and away from a recession?

Is USA Today/Gallup really outside the margin of error re: Obama? Unless they've started reporting on bases above a thousand, I'd be surprised...

It means, JW, that they're ... (Below threshold)

It means, JW, that they're statistically more likely to present polling data that is favourable of Obama and favourable of McCain. Favourability ratings aren't based on preference but on overall opinion, typically--meaning, I could give John McCain a 7 out of 10 (or a "like very well" rating or whatever), but also give Obama an 8 out of 10 (or a "like extremely well"). Not exclusive of one another.

DJ, this is an EXCELLENT po... (Below threshold)

DJ, this is an EXCELLENT post. Great addition information for me to use when I start to deeply handicap this election.

It's possible that the poll... (Below threshold)
Just John:

It's possible that the polls that are favorable to both are pressing the undecideds.

Thanks for the response, wh... (Below threshold)

Thanks for the response, which now makes perfect sense of the results. Great post!

Of obvious interest is ... (Below threshold)

Of obvious interest is the number of people who LIE when they are polled. I have been trying to find someone who has actually been polled for many years. I was polled once several years ago. Failing to find anyone else who has been polled, I started asking a different question; if polled, would you lie.

The answers I got were interesting. A lie seems likely if a)the question was biased or stupid, or b)someone want to cover for a bias.

For example, I asked Hispanics if they were in favor of Obama. The answers I got were highly negative, yet polls show Hispanics favor Obama. Maybe it is a local deal. Hispanics and Blacks compete for the few good jobs around here.






Follow Wizbang

Follow Wizbang on FacebookFollow Wizbang on TwitterSubscribe to Wizbang feedWizbang Mobile


Send e-mail tips to us:

[email protected]

Fresh Links


Section Editor: Maggie Whitton

Editors: Jay Tea, Lorie Byrd, Kim Priestap, DJ Drummond, Michael Laprarie, Baron Von Ottomatic, Shawn Mallow, Rick, Dan Karipides, Michael Avitablile, Charlie Quidnunc, Steve Schippert

Emeritus: Paul, Mary Katherine Ham, Jim Addison, Alexander K. McClure, Cassy Fiano, Bill Jempty, John Stansbury, Rob Port

In Memorium: HughS

All original content copyright © 2003-2010 by Wizbang®, LLC. All rights reserved. Wizbang® is a registered service mark.

Powered by Movable Type Pro 4.361

Hosting by ServInt

Ratings on this site are powered by the Ajax Ratings Pro plugin for Movable Type.

Search on this site is powered by the FastSearch plugin for Movable Type.

Blogrolls on this site are powered by the MT-Blogroll.

Temporary site design is based on Cutline and Cutline for MT. Graphics by Apothegm Designs.

Author Login

Terms Of Service

DCMA Compliance Notice

Privacy Policy