« Breaking News - "Bailout" bill fails in House, Stock plunge | Main | 'Main Street' Not Feeling Championed »

Is There a "Good" Poll?

Looking at Real Clear Politics this morning, I noticed that there is still a range of opinion about the presidential race. Obama supporters will be glad to hear that Rasmussen Tracking, Hotline/FD Tracking, and Gallup Tracking all say Obama is leading McCain by 5 points or more. McCain supporters can be consoled by the fact that Battleground Tracking says McCain is still leading by 2 points. Neither would be happy to hear that trusting the headlines from a "tracking" poll would be about as good an idea as eating a sandwich made in Galveston 17 days ago. The short version of tracking polls is that they are trendy ways for polling groups to get a headline and maybe some attention, but they are simply unproven as empirical analysis. Tracking polls are volatile, depend on rough methodology which tends to oversample some demographics in the interest of speed, and have never been peer-reviewed the way more traditional polling is done. These guys even get cute, and try to put these hash polls together to imitate a real effort to get a sense of the national opinion. And, as I have warned many times before, when a poll does not release its internal data to the public, and conceals any significant part of its methodology and weighting, you should treat the pollster like a Bear Stearns banker. Having said this so long, I hear a lot of people say that they will ignore all polls, while others want to know which poll I think is "best". That, however, is not so easy to answer.

First off, almost all polls have some validity. With very few exceptions, I do believe that polling groups with more than four years of experience in the business will try to follow a standardized methodology, and I also believe that most polls make a determined effort to avoid bias as much as possible. When I say that a poll has bias, therefore, what I mean is that the poll may be accepted for its findings, provided the reader is aware of the poll's history and tendencies. If the poll is consistent, you can compensate for bias and get a generally objective idea of the situation. What's more, if a poll is consistent in its methodology and weightings, then movement by a candidate over time within the same poll can be considered valid to show growing or diminishing strength of support.

So, you don't care about all the details, you just want to know who's winning? Good luck with that, the national polls may not be much real help. That's due to a number of things. First, you should know that the polls taken before the last week of the campaign have no statistical value in predicting the election winner. Also, as I said there is always bias present, so the fact that one poll or many says something does not make it necessarily so (just one reason that both Obama and Mccain have hired private firms to poll for them). And then there is a crucial fact to consider about the election; it's not one race, actually, it's fifty-one races to decide the matter. As Mister Gore realized a bit late in the 2000 election, it is the Electoral vote which determines the presidency.

So, as several people have asked, how do the state polls look? Well, that's a tougher question. First off, you know how I am about transparency in poll internal data, but very few polls make that information public (and some, like Rasmussen, will provide it but only if you pay a fee, which goes against my principles - if you want the publicity of announcing your poll results to the media, you have the moral duty to provide all the supporting data). A significant exception is Survey USA, which is the best of the state polls in following clear NCPP rules. Also, I should warn the reader that in 2004, a lot of state polls were well off the mark. During the last month of the 2004 election, for example, I remember some polls which gave President Bush leads in Michigan and Oregon, and Kerry leads in West Virginia and Florida, none of which turned out to be correct in the actual election. State polls, it should be understood, are smaller budget than the national polls, and often have a much smaller base, which creates a much greater margin of error.


TrackBack

TrackBack URL for this entry:
/cgi-bin/mt-tb.cgi/31941.

Comments (7)

DJ,I'm with you on... (Below threshold)
Baggi:

DJ,

I'm with you on these polls but I was reading a criticism of polls the other day on a different site where an Obama basically said, "Good luck with that" and linked to Real Clear Politics 2004 polls.

Those polls showed quite clearly that all along the average showed Bush leading and low and behold, Bush won.

So my question to you DJ is why can't we make the logical leep that if Bush lead in the RCP average in 2004 and ended up winning, whoever leads in the RCP average in 2008 will end up winning?

Sorry, that should have rea... (Below threshold)
Baggi:

Sorry, that should have read "Obama supporter"

The thing about the 2004 RC... (Below threshold)
DJ Drummond:

The thing about the 2004 RCP averages Baggi, is that 2004 was just one year. And if you look more closely at those polls, you can see that there was disagreement. Marist's last poll had Kerry winning, for example,

http://www.realclearpolitics.com/bush_vs_kerry.html

and so did FOX News, with CNN/USA Today/Gallup and ARG calling it a tie.

In early October, Ap/Ipsos said Kerry had a 4 point lead, and Newsweek said Kerry was ahead for many of their polls.

Also, look at the Bush leads this time in 2004. Time had Bush by 11 in early September, and CBS had Bush by 9 in late September. While you can say the "average" favored Bush, that is not valid in a statistical sense, because there were - looking back - clear outliers which exaggerated movement of support.

Looking back at 2000, the NCPP warns that of the ten major polls for that election, "2 of the 10 polls overstated Gore's vote while 7 overstated Bush. In the 1996 election, 8 out of 9 polls overstated Democrat Clinton",

http://www.ncpp.org/?q=node/20

yet another warning against assuming the headline tells the story. Poll aggregation is not a valid means to determining a candidate's strength.

- continued -

(continued, system won't al... (Below threshold)
DJ Drummond:

(continued, system won't allow too many links)


Looking more closely at 2000, if you check the poll results at election day, you may see a fascinating result.

http://www.pollingreport.com/wh2genT.htm


Voter.com, for example, thought Bush would win the popular vote by 5 percent, and a poll called "Hotline" called it for Bush by 9 points! On October 26, CNN/USA Today/Gallup had Bush up by 13 points! The actual result was a bit different, eh?

By the way, let's go back to that RCP average, and look at the graph of "average" support:

http://www.realclearpolitics.com/Presidential_04/chart3way.html

Notice how much movement there is, and the movement was - this is important - NOT linked to specific causes or actions most of the time.

If you can show me, say, fifteen straight elections where the aggregate poll results matched the actual candidate performance more than 10 days before the election, I will say you have something. But one year of polls which final presentation taken together still missed the actual results by a combined margin greater than their claimed margin of error, well, that does not impress me, and it should not impress you. It reminds me, in a way, of a football player I once saw, who began dancing on his way to the end zone when he reached the 15-yard line. You may recall how that turned out.

Polls are "experience rated... (Below threshold)
Roy Lofquist Author Profile Page:

Polls are "experience rated" - that is, they rely on historical data for their weighting factors. They are predicting the last election. This has proven to be less than reliable.

When an unusual election comes along - this is by far the strangest one I've seen since I started watching in 1952 - they miss by a long way. They totally failed to catch the landslide in 1980 and the 20 million increase in turnout in 2004.

Quit reading "The Racing Form" and watch the race. It is easier on the blood pressure and far more entertaining.

Polls are very much self fu... (Below threshold)

Polls are very much self fulfilling. If the polls (eroneously) averaged 68/29 for McCain the last 2 months, then McCain would eventually develop a REAL 68/29 lead. The pollsters and the "news" media cause the same type of effect (incrementally) in the direction of their guy every time. Then they try to get it right for real just before the election so they can maintain their cred (That'd be crud, to folks like me.).

And dats da name a dat tune.

But one year of po... (Below threshold)
bains:
But one year of polls which final presentation taken together still missed the actual results by a combined margin greater than their claimed margin of error, well, that does not impress me, and it should not impress you.
The constant referral to RCP's aggregate drives me nuts. The inclusion of polling data into an average just because it was done skews results irrevocably. And RCP regularly includes outliers.



Advertisements









rightads.gif

beltwaybloggers.gif

insiderslogo.jpg

mba_blue.gif

Follow Wizbang

Follow Wizbang on FacebookFollow Wizbang on TwitterSubscribe to Wizbang feedWizbang Mobile

Contact

Send e-mail tips to us:

[email protected]

Fresh Links

Credits

Section Editor: Maggie Whitton

Editors: Jay Tea, Lorie Byrd, Kim Priestap, DJ Drummond, Michael Laprarie, Baron Von Ottomatic, Shawn Mallow, Rick, Dan Karipides, Michael Avitablile, Charlie Quidnunc, Steve Schippert

Emeritus: Paul, Mary Katherine Ham, Jim Addison, Alexander K. McClure, Cassy Fiano, Bill Jempty, John Stansbury, Rob Port

In Memorium: HughS

All original content copyright © 2003-2010 by Wizbang®, LLC. All rights reserved. Wizbang® is a registered service mark.

Powered by Movable Type Pro 4.361

Hosting by ServInt

Ratings on this site are powered by the Ajax Ratings Pro plugin for Movable Type.

Search on this site is powered by the FastSearch plugin for Movable Type.

Blogrolls on this site are powered by the MT-Blogroll.

Temporary site design is based on Cutline and Cutline for MT. Graphics by Apothegm Designs.

Author Login



Terms Of Service

DCMA Compliance Notice

Privacy Policy