« News Failure: Getting It Wrong, Setting It Right | Main | BBC is Helping Defeat Obama »

The Weight Room

I wondered about the way the polls came out in the past week. Maybe I'm just a suspicious type, but from what I could tell, a lot of voters had made up their minds before the conventions, and while I think Sarah Palin is going to make a big difference in the election, I did not expect a lot of voters to come over right away; I figure they want to find out more, first. The reason I say this, is that I found it strange to see poll numbers change so quickly. At least part of the answer, I found, was that the results were spun by fooling around with the weighting.

I visited Real Clear Politics, where I noted the new polls. Looking through them, I found two which provided the details on their party affiliation weighting; ABC News/Washington Post, and CBS News. Here's how they cast the last two poll results:

ABC News/Washington Post
Sept. 7 (RV)____________Aug. 21 (RV)
McCain : 46%____________43%
Obama : 47%_____________49%
Republicans: 28%________26%
Democrats: 36%__________36%
Independents: 32%_______33%

I ignored "likely voter" results, because CBS News only tracked registered voters, and I wanted as much apples-to-apples as possible.

Demographics from washingtonpost.com (registration required)


CBS News
Sept. 7 (RV)____________Aug. 19 (RV)
McCain: 46%_____________42%
Obama: 44%______________45%
Republicans: 30.6%______28.8%
Democrats: 36.8%________36.3%
Independents: 32.5%_____35.0%


Both polls increased the weight of Republicans in the new poll. Should it surprise anyone then, that McCain's numbers improved?

Let's play a little game to show how this works. Working back the numbers, it appears that the following matrices of support can be shown using the data provided:

ABC News/Washington Post
August 21 (26% Rep, 36% Dem, 33% Ind)
Republicans: 7% Obama, 88% McCain
Democrats: 87% Obama, 8% McCain
Independents: 49% Obama, 51% McCain
TALLY - 49.31% Obama, 42.59% McCain

September 7 (28% Rep, 36% Dem, 32% Ind)
Republicans: 5% Obama, 92% McCain
Democrats: 87% Obama, 8% McCain
Independents: 46% Obama, 53% McCain
TALLY - 47.44% Obama, 45.60% McCain


CBS News
August 19 (28.8% Rep, 36.3% Dem, 35.0% Ind)
Republicans: 3% Obama, 87% McCain
Democrats: 84% Obama, 4% McCain
Independents: 40% Obama, 45% McCain
TALLY - 45.356% Obama, 42.258% McCain

September 7(30.6% Rep, 36.8% Dem, 32.5% Ind)
Republicans: 3% Obama, 93% McCain
Democrats: 84% Obama, 4% McCain
Independents: 38% Obama, 49% McCain
TALLY - 44.180% Obama, 45.855% McCain

I would point out that I am not making those numbers up - they come from actual poll internals I have been reading, and for this exercise you can see that if you plug them into the weights, you get the results published, more or less. Now, let's take the average weighting of the four weights used, and apply it consistently, and then let's see what happens to the published results:

[averaged weighting 28.35% Rep, 36.0275% Dem, 33.125% Ind]

ABC News/Washington Post
August 21, 49-43 Obama becomes 50-45 Obama
September 7, 47-46 Obama becomes 48-47 Obama

CBS News
August 19, 45-42 Obama becomes 44-41 Obama
September 7, 46-44 McCain becomes 44-44 tie.

- continued -


This is not to say that these 'revised' numbers reflect a more accurate picture of voter support, but it does show that changing the party affiliation weighting can have a significant effect on the published results, especially in the headline which is all that most people read.

The problem with the weighting used in these and other polls, is that there is no science behind the weighting assigned to party identification. ABC/WaPo and CBS are just using whatever affiliation weighting they want to use, on no basis other than they chose to plug in that number. That's fine for fooling around with the settings on a video game when you just want to have some fun, but it is hardly credible for a - purportedly - professional group to do this sort of thing. This kind of squirrelling around with internal controls on a poll is one reason, I think, why polls are so fragmented. I mean, Gallup does its own polls but also teamed up with USA Today. CBS News hires an agency for its own polls, but cooperates with the New York Times for a different poll. Disagreement on key demographic questions would explain that behavior, I think. I had a nice private discussion with one of Gallup's executives back in 2004, and he agreed that the party identification problem is one of the big three for polling groups, largely because there is no consensus on what weighting should be used. Call me silly, but the best metric I have found to use, is the actual voter preferences from national elections.

In 2006, the National Exit Poll showed that 38% of voters considered themselves Democrats, 36% considered themselves Republicans, and 26% considered themselves Independent or supporting another party.

In 2004, the National Exit Poll showed that 37% of voters considered themselves Democrats, 37% considered themselves Republicans, and 26% considered themselves Independent or supporting another party.

In 2002, the National Exit Poll showed that 39% of voters considered themselves Democrats, 38% considered themselves Republicans, and 23% considered themselves Independent or supporting another party.

In 2000, the National Exit Poll showed that 39% of voters considered themselves Democrats, 35% considered themselves Republicans, and 27% considered themselves Independent or supporting another party.

In 1998, the National Exit Poll showed that 39% of voters considered themselves Democrats, 33% considered themselves Republicans, and 28% considered themselves Independent or supporting another party.

Those numbers look rather consistent to me, suggesting we could take an average of the last ten years of elections and get a practical idea of what to expect. Here's how that shaped up:

Democrats: From 37 to 39 percent, average over the last ten years is 38.4%
Republicans: From 35 to 38 percent, average over the last ten years is 35.8%
Independents: From 23 to 28 percent, average over the last ten years is 26.0%


These numbers come from actual election exit polls from verified voters, and they demonstrate consistency over a decade of choice. They demonstrate that change does occur, but in small amounts and over time. Neither the Democrats nor the Republicans should fear becoming a weak party anytime soon, nor should they fool themselves into thinking that their opponents are about to become irrelevant.

I cannot resist, however, applying these standardized party affiliations from known elections to the ABC/WaPo and CBS polls. Here's how that worked out:

ABC News/Washington Post
August 21: 49-43 Obama original report. Historical weights indicate 49-48 Obama.
September 7: 47-46 Obama original report. Historical weights indicate 50-47 McCain .

CBS News
August 19: 45-42 Obama original report. Historical weights indicate 44-44 tie.
September 7: 46-44 McCain original report. Historical weights indicate 48-43 McCain.

I'm not saying you should count on those numbers, but I do think there's better evidence for them.


TrackBack

TrackBack URL for this entry:
/cgi-bin/mt-tb.cgi/31492.

Comments (8)

Interesting. It means that ... (Below threshold)
oy:

Interesting. It means that the second poll exactly coincides with the Monday and Tuesday Gallup tracking poll results, and that the first is close to the Gallup non-tracking poll.

What do you think of Gallup, generally?

Great job, DJ. I learned a... (Below threshold)

Great job, DJ. I learned a while ago that if you swing the poll about three percentage points toward the Republican candidate, you'll get a more accurate reflection of popular sentiment in these kinds of polls.

Of course, it won't matter what you prove with the numbers. If McCain wins, the left will scream "stolen election" simply because the actual results don't mirror their skewed pre-election projections.

Worse still, pundits the world over will describe a majority of our electorate as "racists" because of the discrepancy -- as though poor pre-election sampling couldn't account for Obama's missing margin better than the tired, post-event rationalization that random people often lie to anonymous pollsters about who they're planning to pull the lever for come November.

I'm sure there are some people out there who want the social credit for supporting the black candidate without actually voting for him/her, but poor weighting/sampling of party affiliation (and gender) is a better explanation. And unfortunately for the Obama/Biden campaign, the voters that go to great lengths to appear in sync with identity politics (and whatever grievance rules the day) are unlikely to be on our side in this election, as indicated by Obama's relative underachievement in the primaries.

So all of this bodes well for the Republicans, if the trend continues.

I noticed the Obama people ... (Below threshold)

I noticed the Obama people and Halperin at Time are giddily hyping the as of right now emargoed WJ/NBC Poll.

I wonder what sample size and sample dates were used to jimmy the numbers to get them so excited.

McCain voted the same as Ob... (Below threshold)
Joe:

McCain voted the same as Obama 50 percent of the time in 2006 Roll Call Votes.

http://www.CongressOverTime.com/votes/s2006.xml

(Click the column headings in the detailed table.)

I recall in the last 2 pres... (Below threshold)
mike:

I recall in the last 2 presidentials that polling firms have done this to expand the "bounce." This also makes it appear that the bounces dissipate faster over time.

Do you think that there are purposeful reasons why the polling firms do this?

Joe, let's say there are tw... (Below threshold)
DJ Drummond:

Joe, let's say there are two umpires. They both get 90+% of their calls right, and make the same number of mistakes. But one of them, his mistakes are ball/strike calls right on the edge, but he always gets home runs and put-outs right. The other gets more of the iffy stuff right, but he blows calls which decide the game.

Would you claim they are both the same caliber of umpire?

I realize this is a lot of ... (Below threshold)

I realize this is a lot of work DJ and I appreciate the effort.

Please keep these posts coming on a regular basis as they are very informative and I trust you to call BS on a poorly constructed poll (either way).

I haven't looked into this ... (Below threshold)
Clint:

I haven't looked into this as closely as you have, but I think you may be overlooking a point.

The number of people self-identifying as Republican has actually increased in the last week. This is probably a reflection of the increased excitement of Republicans.

While using the same party-breakdown from week to week would give smaller bounces (NB: Rasmussen does it this way, hence the smaller bounces) it would also mean missing out on the changes in enthusiasm -- changes that are quite real and relevant.

Which way is the "right" way to do it?

Both? Neither?

No idea.




Advertisements









rightads.gif

beltwaybloggers.gif

insiderslogo.jpg

mba_blue.gif

Follow Wizbang

Follow Wizbang on FacebookFollow Wizbang on TwitterSubscribe to Wizbang feedWizbang Mobile

Contact

Send e-mail tips to us:

[email protected]

Fresh Links

Credits

Section Editor: Maggie Whitton

Editors: Jay Tea, Lorie Byrd, Kim Priestap, DJ Drummond, Michael Laprarie, Baron Von Ottomatic, Shawn Mallow, Rick, Dan Karipides, Michael Avitablile, Charlie Quidnunc, Steve Schippert

Emeritus: Paul, Mary Katherine Ham, Jim Addison, Alexander K. McClure, Cassy Fiano, Bill Jempty, John Stansbury, Rob Port

In Memorium: HughS

All original content copyright © 2003-2010 by Wizbang®, LLC. All rights reserved. Wizbang® is a registered service mark.

Powered by Movable Type Pro 4.361

Hosting by ServInt

Ratings on this site are powered by the Ajax Ratings Pro plugin for Movable Type.

Search on this site is powered by the FastSearch plugin for Movable Type.

Blogrolls on this site are powered by the MT-Blogroll.

Temporary site design is based on Cutline and Cutline for MT. Graphics by Apothegm Designs.

Author Login



Terms Of Service

DCMA Compliance Notice

Privacy Policy