« The Knucklehead of the Day award | Main | Obama says Palin's family off-limits »

How Polls Work - And How They Do Not Work

In an earlier article, a reader noted that he was suspicious of CNN's "Poll of Polls". Frankly, he was right to do so. What CNN, Real Clear Politics, and other media outlets do, is to take poll releases from various polling groups in a certain time range, and aggregate them to find a sort of consensus. That is a very bad idea, however, if your intention is to accurately reflect the opinion of the general public. It's a bit like saying that if you take everyone's favorite version of spaghetti and mix them all together, you will get a really great-tasting batch of spaghetti. The odds are you will get a mess which won't be worth the effort, and that's what happens when you mix poll results. Polls, it should be noted, are the product of the groups and agencies which create them, and reflect a specific methodology which is usually similar to that used by other polling groups, but not exactly. That difference is why the two results cannot be mixed with confidence, and the more polls that are mixed, the less reliable would be the resulting report. Polls are best described as snapshots of opinion on one question at one specific point in time and place, not even movies much less reality. An example of this can be seen by looking at the most reliable of agencies, the Gallup Organization.

On August 24, Gallup showed Obama and McCain tied at 45% support each (with 10% undecided). But three days later, Gallup showed Obama with a six-point lead, 48% to 42%, and today Obama still leads by six points, 49% to 43%. The Gallup people seem to be claiming that Obama has increased his support while McCain has lost some. However, a look at the history of convention "bounces" would warn against making such an assumption. If history holds true, for example, McCain should enjoy a similar "bounce" after the GOP convention, yet this would not be a valid indicator of a stronger campaign, necessarily, but merely the effect of the focus from the convention. Even within the stable methodology of a well-respected poll, therefore, it is not valid to assume that changes in polling are necessarily indicators of a weaker or stronger election position.

With that said, it is important to consider other polling results. An August 30 release from Zogby, for instance, shows McCain at 47% support and Obama at 45%, at the same time that Gallup showed Obama ahead by six and holding steady support. How can two major polls say different things? Well, part of it is the fact that the variance between the polls is only eight points, which is equivalent to a four-point margin of error between them. If either poll had a respondent pool of 500 people or less, that disparity is within the standard deviation for such polling. That's the mathematical point. What else needs to be considered, is that even where the same polling group takes polls, because of the randomness of the respondent pool, this means by definition that the people in different polls are not the same ones who answered the earlier poll, and even when conditions are the same and the same careful rules are applied, different people will probably produce different results. Also, as I have noted before, there is a clear bias in favor of or against the people in focus of polls by the polling groups. As a result, a composite of poll results cannot, by its character, produce an accurate or trustworthy result. At best, one of the polls might accurately reflect the opinion at that moment, but even if that is the case, mixing the results from all polls would only insure that the aggregate result was wrong to some degree.

Polls are not effective predictors of elections. The losing candidates of many past general and primary campaigns can point to any number of polls which indicated they would win. A candidate who trusts polls to indicate the precise amount and nature of their public support, is following the path of Tom Dewey. That is not to say, however, that polling is worthless. When a poll follows consistent methodology and asks consistent questions, and weights its demographics according to Census norms and the National Council on Public Polls standards (or the AAPOR), the results can be applied to illustrate trends and voter interest in key issues. Also, while the public is familiar with the polls which get published in their newspapers and favorite news sites, it should be noted that both the Obama and McCain campaigns have paid for private polling firms to identify target demographics and battleground states. Obama has spent almost 19 million dollars on private polling, while McCain has spent about 770 thousand dollars. It would fair to assume, I think, that these agencies are far more detailed in their questions and specific in their target demographics. So it may also be assumed from the amount of money spent, that there is significant confidence in the value of effective polling. The real question, as yet uncertain in its answer, is how to know which questions, which method, and which weighting is valid for the real condition.


TrackBack

TrackBack URL for this entry:
/cgi-bin/mt-tb.cgi/31264.

Comments (10)

All those Democrats out the... (Below threshold)
Herman:

All those Democrats out there, with only cell phones, who can't be polled. Got to wonder about the implications.

Oh, by the way, I'm one of them myself!

I always like the methodolo... (Below threshold)

I always like the methodology of the Rasmussen Polls as well as the analysis of Election Project for their accuracy the most. It always seems to me that the Zogby polls are a little flawed for some reason likely related to their own methodology or use of samples. I studied statistics at Portland State University back in 1973 and even conducted some polling for some group class projects, and many variables impact results in an election such as those most likely to vote.

In 1948, classic polling error only relied on those wealthy enough to own a phone, and wrongly predicted a win by Governor Thomas Dewey for example. The Zogby polls are largely only polls of those who own or use a computer that I am aware of, leaving out many voters of older age who do not use computers or who are lower income and do not own one.

Good article DJ. These post... (Below threshold)

Good article DJ. These posts on polls have been interesting.

Mainstream Political Pol... (Below threshold)
Smack MacDougal:

Mainstream Political Polling is UNSCIENTIFIC

--------------------------------------------------------------------------------------------------

A Presidential race is not one race, but 50!


Why?


The Electoral College.

How the Electoral College Works

Most states have a "winner-take-all" system of electoral votes.

This means that the guy with the most popular votes in enough electoral districts gets all the electoral votes for that state.

Yet some states have a "Split Elector System" and let the winner of an elector district to keep that electoral college vote.

How Scientific Polling Works

Say you want to have the results of a poll that you can project to the universe of voters and be right within plus or minus 5%.

Also, say you want your confidence to be 95%, that is, 95 out of 100 times, you will be right within plus or minus 5%.

To get a +/- 5% margin of error, you must complete 376 questionnaries where each respondent interprets exactly each question in the same way.

Moreso, each respondent must come from a valid bounded region, say an Electoral College District.

So to poll scientifically, a pollster would have to complete 50 polls times the required number of completed surveys for a desired margin of error within a bounded region.


Additionally, each demographic division requires a two-fold increase in the required completed questionnaires within the same Survey, e.g., Democrats vs Republicans, women vs men, etc.

Here's an example:


To project to the universe of voters who will elect electors, you would need to poll 376 x 50 = 18,800 surveys completed on the SAME DAY!


(Note: each 376 would have to come from one of the Electoral Districts in each of 50 US states.)


And if you wanted to know how Democrats will vote or voted you would need 376 x 2 x 50 = 37,600 completed surveys on the SAME DAY!

Also, some states allow for splitting of electors, for each of these states, you would have to conduct surveys in each elector district for each elector. So the number of completed surveys at +/-5% ME = 376 x some number greater than 50.

To know exactly who shall win, you would need to poll 270 Electoral Districts whose voters choose the same candidate.

This means you would need to complete 270 x 376 = 101,520 questionnaries!

Wait, there's more. You would have to do this on the same day around the same time!

Political Polls Give FAKE RESULTS, First Cause

All of the Major Pollsters (you know who they are) never conduct polls in the Scientific Method detailed above, thus their polls are unscientific and their claims of prediction utterly false.

Don't believe me?

Ask any statistical mathematics professor.

To get results that you can project to the universe of registered voters, you must conduct a poll in each Electoral College District.

Political Polls Give FAKE RESULTS, Second Cause

A poll cannot be conducted over several days.


Why?


The events of the next news day can change the beliefs and wants of any person and all persons.


Pollsters spread out the surveys over several days, which is typical of them.


Each person polled has a different perspective based upon different knowledge.


Some new repondents will base their choice upon the latest news. Others, who have already responded, might change their choice -- but they cannot since they will not be asked again!


The world is chaotic.


A tumbling DOW Jones Industrial Average, a war breaking out in Georgia, a hurricane slamming the Gulf of Mexico, all of these events change the beliefs, wants and lives of many persons, from the moment of the event going forward.


If you poll over several days, all you are doing is collecting the opinions of those polled and thus, you CANNOT project these opinions as a statistical measure upon the entire universe of voters.


In other words, Polling Services and Mainstream Media have defrauded you for decades.

Political Polls Give FAKE RESULTS, Third Cause


The results of a Poll are good only for the day they are taken!


A dirty secret never mentioned by pollsters is that the results of a Poll, if done right (taken scientifically) are good only for the day they are taken!


What?


That's right.

Assuming you conduct your poll accurately and without bias within any of the questions as well as how you derive your sample, then you can make claims about the universe of voters only on the day that you polled them.

As mentioned above, since new events can change the beliefs, wants and opinions of persons, the results from any poll are good only for the day they are taken.

Any pollster who claims that poll results are projectable upon the universe of voters and hold true for the following day or any day thereafter, then that pollster is lying.

Polls do not carry meaning from the day after you take them nor do they hold true through the next poll.

Once again, pollsters and the mainstream media that carries polls has been consistently defrauding Americans over the truth of polling.

So Why Do Polls Work Then?

Published Polling within Mainstream Media establishes what is known by insiders as Predictive Programming.

When a lie is told often enough, a mass of persons accept the lie in short order.

Stanley Milgram proved this years ago with his study reported as Obedience to Authority.

In short, no one wants to shout that the Emperor has no clothes, thus everyone goes along with the story told given by Political Polling in Mainstream Media.

In other words, the Media or their Backers push the Voting Public to pick the candidate they want.

Unwritten rule - if your co... (Below threshold)
DJ Drummond:

Unwritten rule - if your comment is as long as my article, you should post your own blog, don't eat up the space for your own hype.

Clue part 2, Smacky - most of us knew the facts about polling, though we have - thankfully - managed to determine our own conclusions w/o resorting to the handy-dandy Conspiracy Theory mindset.

Seriously Smack-boy, get professional help.

Good article. Personally, ... (Below threshold)
GarandFan:

Good article. Personally, the only poll I take creedence in will be the one held on Nov 4.

Polls can also be skewed by the way the question is framed.

I don't believe you, Paul. ... (Below threshold)
hyperbolist:

I don't believe you, Paul. If Rasmussen was polling only voters or households with computers or internet access, that would have to be explicit. They would have to say "X% of eligible voters with internet access/computer" or something like that. If the base of a survey is limited in such a significant way, it has to be reported or the results are (at best) misleading.

As for income, any poll that purports to be nationally representative would have to be sampled on the most current census data. So while a national poll's actual numbers might skew towards wealthier, older people (as they inevitably do--retired old folks are most likely to sit through a phone or internet survey), its weighted numbers would reflect something closer to national representation.

Which leads me to a question for DJ: what do you dislike about weighting, given a sufficiently large set of data (say, 1,000 survey completions)?

I haven't seen any j... (Below threshold)
Larry:


I haven't seen any justification on Real Clear for why they do what they do the way they do it. That site does seem to be very popular with pundits and news types for whatever reason. What do you think of Jay Cost and his analysis?

Larry

Polls are used and manipula... (Below threshold)
epador:

Polls are used and manipulated just as the Public Media to influence opinions and actions by voters.

That's my opinion, I won't waste smack space trying to justify it.

You amuse DJ Drummond.... (Below threshold)
Smack MacDougal:

You amuse DJ Drummond.

When challenged about your false beliefs, your cognitive dissonance whips you into defense mode.

Your childish rant makes for much laughter.

And your projection about "Conspiracy Theory" adds the icing.

You're a psychologist's wet dream.




Advertisements









rightads.gif

beltwaybloggers.gif

insiderslogo.jpg

mba_blue.gif

Follow Wizbang

Follow Wizbang on FacebookFollow Wizbang on TwitterSubscribe to Wizbang feedWizbang Mobile

Contact

Send e-mail tips to us:

[email protected]

Fresh Links

Credits

Section Editor: Maggie Whitton

Editors: Jay Tea, Lorie Byrd, Kim Priestap, DJ Drummond, Michael Laprarie, Baron Von Ottomatic, Shawn Mallow, Rick, Dan Karipides, Michael Avitablile, Charlie Quidnunc, Steve Schippert

Emeritus: Paul, Mary Katherine Ham, Jim Addison, Alexander K. McClure, Cassy Fiano, Bill Jempty, John Stansbury, Rob Port

In Memorium: HughS

All original content copyright © 2003-2010 by Wizbang®, LLC. All rights reserved. Wizbang® is a registered service mark.

Powered by Movable Type Pro 4.361

Hosting by ServInt

Ratings on this site are powered by the Ajax Ratings Pro plugin for Movable Type.

Search on this site is powered by the FastSearch plugin for Movable Type.

Blogrolls on this site are powered by the MT-Blogroll.

Temporary site design is based on Cutline and Cutline for MT. Graphics by Apothegm Designs.

Author Login



Terms Of Service

DCMA Compliance Notice

Privacy Policy