The Weight Room

I wondered about the way the polls came out in the past week. Maybe I’m just a suspicious type, but from what I could tell, a lot of voters had made up their minds before the conventions, and while I think Sarah Palin is going to make a big difference in the election, I did not expect a lot of voters to come over right away; I figure they want to find out more, first. The reason I say this, is that I found it strange to see poll numbers change so quickly. At least part of the answer, I found, was that the results were spun by fooling around with the weighting.

I visited Real Clear Politics, where I noted the new polls. Looking through them, I found two which provided the details on their party affiliation weighting; ABC News/Washington Post, and CBS News. Here’s how they cast the last two poll results:

ABC News/Washington Post
Sept. 7 (RV)____________Aug. 21 (RV)
McCain : 46%____________43%
Obama : 47%_____________49%
Republicans: 28%________26%
Democrats: 36%__________36%
Independents: 32%_______33%

I ignored “likely voter” results, because CBS News only tracked registered voters, and I wanted as much apples-to-apples as possible.

Demographics from washingtonpost.com (registration required)

CBS News
Sept. 7 (RV)____________Aug. 19 (RV)
McCain: 46%_____________42%
Obama: 44%______________45%
Republicans: 30.6%______28.8%
Democrats: 36.8%________36.3%
Independents: 32.5%_____35.0%

Both polls increased the weight of Republicans in the new poll. Should it surprise anyone then, that McCain’s numbers improved?

Let’s play a little game to show how this works. Working back the numbers, it appears that the following matrices of support can be shown using the data provided:

ABC News/Washington Post
August 21 (26% Rep, 36% Dem, 33% Ind)
Republicans: 7% Obama, 88% McCain
Democrats: 87% Obama, 8% McCain
Independents: 49% Obama, 51% McCain
TALLY – 49.31% Obama, 42.59% McCain

September 7 (28% Rep, 36% Dem, 32% Ind)
Republicans: 5% Obama, 92% McCain
Democrats: 87% Obama, 8% McCain
Independents: 46% Obama, 53% McCain
TALLY – 47.44% Obama, 45.60% McCain

CBS News
August 19 (28.8% Rep, 36.3% Dem, 35.0% Ind)
Republicans: 3% Obama, 87% McCain
Democrats: 84% Obama, 4% McCain
Independents: 40% Obama, 45% McCain
TALLY – 45.356% Obama, 42.258% McCain

September 7(30.6% Rep, 36.8% Dem, 32.5% Ind)
Republicans: 3% Obama, 93% McCain
Democrats: 84% Obama, 4% McCain
Independents: 38% Obama, 49% McCain
TALLY – 44.180% Obama, 45.855% McCain

I would point out that I am not making those numbers up – they come from actual poll internals I have been reading, and for this exercise you can see that if you plug them into the weights, you get the results published, more or less. Now, let’s take the average weighting of the four weights used, and apply it consistently, and then let’s see what happens to the published results:

[averaged weighting 28.35% Rep, 36.0275% Dem, 33.125% Ind]

ABC News/Washington Post
August 21, 49-43 Obama becomes 50-45 Obama
September 7, 47-46 Obama becomes 48-47 Obama

CBS News
August 19, 45-42 Obama becomes 44-41 Obama
September 7, 46-44 McCain becomes 44-44 tie.

– continued –

]]>< ![CDATA[

This is not to say that these ‘revised‘ numbers reflect a more accurate picture of voter support, but it does show that changing the party affiliation weighting can have a significant effect on the published results, especially in the headline which is all that most people read.

The problem with the weighting used in these and other polls, is that there is no science behind the weighting assigned to party identification. ABC/WaPo and CBS are just using whatever affiliation weighting they want to use, on no basis other than they chose to plug in that number. That’s fine for fooling around with the settings on a video game when you just want to have some fun, but it is hardly credible for a – purportedly – professional group to do this sort of thing. This kind of squirrelling around with internal controls on a poll is one reason, I think, why polls are so fragmented. I mean, Gallup does its own polls but also teamed up with USA Today. CBS News hires an agency for its own polls, but cooperates with the New York Times for a different poll. Disagreement on key demographic questions would explain that behavior, I think. I had a nice private discussion with one of Gallup’s executives back in 2004, and he agreed that the party identification problem is one of the big three for polling groups, largely because there is no consensus on what weighting should be used. Call me silly, but the best metric I have found to use, is the actual voter preferences from national elections.

In 2006, the National Exit Poll showed that 38% of voters considered themselves Democrats, 36% considered themselves Republicans, and 26% considered themselves Independent or supporting another party.

In 2004, the National Exit Poll showed that 37% of voters considered themselves Democrats, 37% considered themselves Republicans, and 26% considered themselves Independent or supporting another party.

In 2002, the National Exit Poll showed that 39% of voters considered themselves Democrats, 38% considered themselves Republicans, and 23% considered themselves Independent or supporting another party.

In 2000, the National Exit Poll showed that 39% of voters considered themselves Democrats, 35% considered themselves Republicans, and 27% considered themselves Independent or supporting another party.

In 1998, the National Exit Poll showed that 39% of voters considered themselves Democrats, 33% considered themselves Republicans, and 28% considered themselves Independent or supporting another party.

Those numbers look rather consistent to me, suggesting we could take an average of the last ten years of elections and get a practical idea of what to expect. Here’s how that shaped up:

Democrats: From 37 to 39 percent, average over the last ten years is 38.4%
Republicans: From 35 to 38 percent, average over the last ten years is 35.8%
Independents: From 23 to 28 percent, average over the last ten years is 26.0%

These numbers come from actual election exit polls from verified voters, and they demonstrate consistency over a decade of choice. They demonstrate that change does occur, but in small amounts and over time. Neither the Democrats nor the Republicans should fear becoming a weak party anytime soon, nor should they fool themselves into thinking that their opponents are about to become irrelevant.

I cannot resist, however, applying these standardized party affiliations from known elections to the ABC/WaPo and CBS polls. Here’s how that worked out:

ABC News/Washington Post
August 21: 49-43 Obama original report. Historical weights indicate 49-48 Obama.
September 7: 47-46 Obama original report. Historical weights indicate 50-47 McCain .

CBS News
August 19: 45-42 Obama original report. Historical weights indicate 44-44 tie.
September 7: 46-44 McCain original report. Historical weights indicate 48-43 McCain.

I’m not saying you should count on those numbers, but I do think there’s better evidence for them.