« Pro-Obama Push Polling Begins | Main | The Magical Disappearing SNL Skit! »

Bwa Ha Ha

Lorie pointed out that CBS has a new poll, so I went and had a look, and it's funnier than the comics. Sure, I generally like CBS because they have a consistent methodology, and because they always publish their questions and party breakdown. This allows me to reverse their calculations and see how they got their results.

First off, it's more fun if you play along at home, so here's the poll address.

OK, now let's keep it simple and start with page 14, the last page. Let's read together the last two sentences in that poll:

"Every registered voter is included in the likely voter model, and is assigned a probability of voting, which is used to calculate the likely voter results. The sum of these probabilities is the effective number of likely voters."

Pfft-pfft-bfft-bwaaaahahahahahahahahahahahahahahahaha!!!!

In those two sentences, the folks at CBS are admitting that they MADE UP THEIR STATISTICS ON LIKELY VOTERS!!! There are no real voters who were asked if they were likely to vote in this poll - not even one! Remember, these polls are how the media tells us what they claim we are thinking, and here at the end is a disclaimer saying that at least a portion of their major findings is fabricated. Just something to keep in mind as we move along, nothing to see here, move along.

Also on page 14, just above that comedy about CBS making up their 'likely voter' database, is the list of respondents. Let's mark that, and total up the participants and their proportions:

...................... Unweighted ....... Weighted
Total Respondents .... 957
Registered Voters .... 875 ................ 821
Republicans (RV) ..... 248 (28.3%) ...... 233 (28.4%)
Democrats (RV) ....... 320 (36.6%)....... 311 (37.9%)
Independents (RV)..... 307 (35.1%)....... 277 (33.7%)

Why is this important? Because let's go look at page 8 and look at the support by party affiliation:

...................Obama ......... McCain
Republicans........ 7%............ 86%
Democrats ......... 85%........... 10%
Independents ...... 39%........... 44%

CBS released a poll headline, remember, saying Obama was leading McCain 47% to 43% among Registered Voters. So let's apply the math of the weighting and see how they did:

Obama:
Republicans: 7% at .284 = 1.99%
Democrats: 85% at .379 = 32.22%
Independents: 39% at .337 = 13.14%
Total : 1.99 + 32.22 + 13.14 = 47.35%

McCain:
Republicans: 86% at .284 = 24.42%
Democrats: 10% at .379 = 3.79%
Independents: 44% at .337 = 14.83%
Total: 24.42 + 3.79 + 14.83 = 43.04%

OK, now we have to stop here and ask if those number make sense. Well, for one thing, I like that CBS tells us the exact unweighted numbers. It tells us, for example, that 89.55% of the people they reached said they were registered to vote, and we know that the poll was done over the weekend by telephone, and that the total respondent pool was below 1,000 people. Looking at the detail on page 8, we also know that urban areas figured heavily in the poll (look at the white vote, and the conservative/moderate/liberal breakdown of support). By the way, the claim at the bottom of the page by CBS that the poll has an overall margin of error of only "three percent" is not statistically true, as there are less than 900 valid respondents; the actual MOE would be 3.5 percent.

Why do I think the actual numbers are better for McCain? Let's look again at those specific supports by party affiliation and reline them.

BASE SUPPORT (Republicans for McCain, Democrats for Obama)
McCain 86%, Obama 85%, McCain +1

COUNTER-PARTY SUPPORT (Republicans for Obama, Democrats for McCain)
McCain 10%, Obama 7%, McCain +3

INDEPENDENT SUPPORT
McCain 44%, Obama 39%, McCain +5

Huh.

Well, there's one way that Obama comes out on top and CBS told you on page 14. But what happens if we use the historical party affiliation from, say, the last federal election in 2006? Let's see:

Democrats: 38.5%
Republicans: 35.5%
Independents 26.0%

Obama:
Democrats: 85% at .385 = 32.73%
Republicans: 7% at .355 = 2.49%
Independents: 39% at .260 = 10.14%
Total : 32.73 + 2.49 + 10.14 = 45.36%

McCain:
Democrats: 10% at .385 = 3.85%
Republicans: 86% at .355 = 30.53%
Independents: 44% at .260 = 11.44%
Total: 3.85 + 30.53 + 11.44 = 45.82%

Well within the margin of error, to be sure, but a very different picture, also plainly to be seen.

The lessons are the same as always: Pay attention to the details, note trends within polls rather than aggregating disparate poll reports, and never forget that every media outlet is selling a story, and every poll is bought by someone for a specific purpose. The election is something far different from any poll, and we must not let ourselves be dismayed or misled by what anyone tells us in the media.


TrackBack

TrackBack URL for this entry:
/cgi-bin/mt-tb.cgi/32119.

Listed below are links to weblogs that reference Bwa Ha Ha:

» Sierra Faith linked with Is the Race Really Tied?

» MERE HEALING linked with The numbers game.

Comments (26)

Glad to see this poll has l... (Below threshold)
Pam:

Glad to see this poll has lifted you out of your funk.

His Peace, His Blessings!

Very interesting.H... (Below threshold)
Art:

Very interesting.

Have you done the same analysis for any other polls?

Quite a few, Art.G... (Below threshold)
DJ Drummond:

Quite a few, Art.

Guess you're new here?

I know you've said in the p... (Below threshold)
Ray H.:

I know you've said in the past the state polls are really hard to peg, but even though your numbers really do make sense to me, all the major networks are claiming an Obama landslide is going to occur from an electoral perspective. DJ, do you have any thoughts on this from your stats perspective? I'm not a fan really of either candidate, but really think Obama and a democratic dominated congress is not a good thing for our country.

Well Ray, I am still workin... (Below threshold)
DJ Drummond:

Well Ray, I am still working on some numbers, but you can get started on state polls at this article:

http://stolenthunder.blogspot.com/2008/10/those-state-polls-preface.html

This is just CBS's way of s... (Below threshold)
LaMedusa:

This is just CBS's way of saying, "We got nuthin', so we gotta make up sumthin'. What numbers sound feasible that will kiss the dems as*?"

How unfortunate that a lot of people ride on these numbers as current activity that is "newsworthy". Details, details...Thanks for the heads up, DJ.

Thanks DJ, I'll keep an eye... (Below threshold)
Ray H.:

Thanks DJ, I'll keep an eye out for your analysis.

DJI claim absolute... (Below threshold)
JFO:

DJ

I claim absolutely no expertise as far as polling is concerned. I have an observation and a question.

If most, if not all, accepted polls show trends one way or the other it seems to me that the trend cannot be ignored. It just seems like too much evidence to me.

My question: How is the factor of age taken into account here. I know that there are many young folks who are excited about the race and want to be involved. Yet, most probably have cell phones rather than land lines. So how is all that factored in, if at all?

Great Job DJ. I am register... (Below threshold)
MPR:

Great Job DJ. I am registered to post comments at some other popular blogs and I often send other commenters over here to read your analysis of these skewed polls. Thanks again.

JFO and DJ -What a... (Below threshold)
JLawson:

JFO and DJ -

What about folks using VOIP services like Vonage? One of the main reasons we went to that was we didn't want to wait six weeks to get our phone hooked up by Ma Bell when we moved (cable TV/Broadband was there the day we moved...) and since then we've had NO telemarketers or survey calls of any sort, where we'd been having three or four a week before that.

JFO, good afternoon. You h... (Below threshold)
DJ Drummond:

JFO, good afternoon. You have hit on one of the great problems facing pollsters; how to reach a significant demographic. The people using cell phones, actually, can be reached in much the same way that landline phones can be reached, because RDD methodology does not distinguish between the lines used. The problem is, that people out of the home are much less likely to be willing to take a ten-to-fifteen minute call while they are in the car, at work, or some other place here their attention is limited for polling. At this time there is no good answer.

I would note, by the way, that cell phones are in such common use, that reaching all the phone users would not be likely to impact demographics very much. All ages use cell phones now, albeit for different reasons. It occurs to me, as well, that verification could be tricky, as well, because a lot of elementary and middle school kids carry cell phones now, given by their parents to stay in touch when necessary. We may soon need to ask, "Are You Polling a 5th Grader?"

I love your poll analyses. ... (Below threshold)
Carol:

I love your poll analyses. I expect the media outlet polls to be biased, but can you speak to the Rasmussen (Obama +8) and Gallup (Obama +9) polls? I thought they were considered pretty reputable, and they are showing big leads for Obama -- even bigger than the CBS poll. Thanks!

Carol, Rasmussen is a merce... (Below threshold)
DJ Drummond:

Carol, Rasmussen is a mercenary. As for Gallup, they change their affiliation weights, I am waiting for the Oct 3 details to see where things really stand.

Two things bother me about ... (Below threshold)

Two things bother me about the polling this cycle.

First, the polling has been unusual. Normally, since I've been politically aware, the Democrats tend to lead national votes by 10-15%, until about a month out from the election. At that point, the race tends to narrow. This cycle, it's been the opposite: close all the way along, and now starting to widen out. When long-standing historical patterns change, it makes me wonder if something fundamental has actually changed, or if there is a surface political change attempting to appear as a fundamental shift. Given the increasingly apparent media bias — Who ever thought we'd look back on the Rathergate election as reasonably even-handed? — and the small poll sizes, I suspect that there's less reality here than fantasy.

But that leads to the second problem. I am fairly concerned that, if Obama loses, there will be significant rioting, and a large part of that will be based on the idea that, given what the polls said heading in, a McCain victory could only be because of corruption. That would do vast damage to what remains of trust in our civil institutions, which does not bode well for society as a whole.

If the polls are really pre... (Below threshold)
Burt:

If the polls are really predictors, wouldn't we be looking at Ron Paul at the top of the Republican ticket?

DJ, can you explain what yo... (Below threshold)
Carol:

DJ, can you explain what you mean by "Rasmussen's a mercenary?" I'm a bit dense and don't understand.
Thanks!

A bit of pique, Carol. Sco... (Below threshold)
DJ Drummond:

A bit of pique, Carol. Scott Rasmussen will not release internal data nor his weighting, except to those who first pay for his "premium" service. To me, it's like a student demanding extra credit from the professor just for showing his work, before there is any way to know if the work was properly done at all.

Hence, I dub him a mercenary.

Dj, I'm glad you paid atten... (Below threshold)
JoeC:

Dj, I'm glad you paid attention during Probability and Statistics class. I though it was interesting but made my head hurt. "Can I go outside and play another game?" Now *I* wish I HAD paid more attention.

DJ:Rasmussen does ... (Below threshold)
sam:

DJ:

Rasmussen does not reveal weightings even to his premium customers, especially for any of his state polls.

He does provide crosstabs (x% Dems supporting Obama, Y% Reps supporting McCain etc.), but he does not reveal how many Reps or Dems in his sample.

FWIW.

In that case sam, he is not... (Below threshold)
DJ Drummond:

In that case sam, he is not merely a mercenary but essentially dishonest.

If the details are hidden, the poll cannot be considered valid.

No exceptions.

JeffLook a the pol... (Below threshold)
Larry:

Jeff

Look a the polls today. It is narrowing again.

Q4, in the middle of page t... (Below threshold)
ORyan:

Q4, in the middle of page ten asks respondents how likely they are to vote in 2008 election.

Traditionally don't they ask about prior voting history to determine LV models? I would guess they also used demographic info in the LV calculation as well.

Anyway, I actually think the way they did this is better. What polls have done in the past is set up a cut off point based on the probability that an individual will vote. So, in the LV model everybody beneath an arbitrary point is excluded. I take their statement to mean, if somebody has a 30% chance of voting in their LV model then they add .3 of a vote onto one candidates total instead of completely disregarding the respondents choice if they are beneath the cutoff point. I think this makes more sense than polls in the past. Any time a pollster moves from RV to LV there is a leap of faith because it is hard to know whether somebody will vote or not.

Dude, my eyes glazed over a... (Below threshold)

Dude, my eyes glazed over after the first set of stats. Summarize!

The point, ORyan, is that C... (Below threshold)
DJ Drummond:

The point, ORyan, is that CBS admitted that a major part of their poll was artificially created, that they did not use actual respondent information to directly tally the LV numbers. It's a critical flaw in methodology, and goes directly against NPCC guidelines.

Timmer, the numbers establish the conclusions. You need to learn to comprehend the data in context, "dude".

DJ, the end point of your a... (Below threshold)
DougS:

DJ, the end point of your analysis is basically the same as Jay Cost's at RCP: that this is a very close race, and has been a very close race all along, no matter what the polls say. This agrees with what my instincts tell me right now, too.

I just find it interesting to see Cost take that line of analysis, since one of RCP's trademarks is trying to make sense of the latest polls. But their own campaign blogger seems to be adamant in his refusal to get caught up in the gyrations of polls that, as you say, are flawed.

Can we ever trust any polls... (Below threshold)
Spurwing Plover:

Can we ever trust any polls taken by the COMMUNISTS,BROADCASTING SYSTEM and the EVIL LYING EYE OF SATAN?




Advertisements









rightads.gif

beltwaybloggers.gif

insiderslogo.jpg

mba_blue.gif

Follow Wizbang

Follow Wizbang on FacebookFollow Wizbang on TwitterSubscribe to Wizbang feedWizbang Mobile

Contact

Send e-mail tips to us:

[email protected]

Fresh Links

Credits

Section Editor: Maggie Whitton

Editors: Jay Tea, Lorie Byrd, Kim Priestap, DJ Drummond, Michael Laprarie, Baron Von Ottomatic, Shawn Mallow, Rick, Dan Karipides, Michael Avitablile, Charlie Quidnunc, Steve Schippert

Emeritus: Paul, Mary Katherine Ham, Jim Addison, Alexander K. McClure, Cassy Fiano, Bill Jempty, John Stansbury, Rob Port

In Memorium: HughS

All original content copyright © 2003-2010 by Wizbang®, LLC. All rights reserved. Wizbang® is a registered service mark.

Powered by Movable Type Pro 4.361

Hosting by ServInt

Ratings on this site are powered by the Ajax Ratings Pro plugin for Movable Type.

Search on this site is powered by the FastSearch plugin for Movable Type.

Blogrolls on this site are powered by the MT-Blogroll.

Temporary site design is based on Cutline and Cutline for MT. Graphics by Apothegm Designs.

Author Login



Terms Of Service

DCMA Compliance Notice

Privacy Policy