« Guns and condoms | Main | Al-Qaeda Bombing of the Islamabad Marriott »

There Is No Alternate Universe

People throw out a lot of strange ideas at times. Some of that is because people can believe some very silly things, and some of that I blame on television and comic books. But it really gets strange when we see fantasy and illusion play out in politics, in the guise of fact. CBS tried to smear President Bush with documents they admitted were fakes, the claimed inexperience of a Vice-Presidential candidate is loudly harangued, while the weaker resume of the other major party's Presidential nominee is ignored, and biased opinion polls are touted as objective sources of news. Truly Alice-in-Wonderland stuff, except that this is the real world, and we dare not risk trusting the Mad Hatter.

Opinion polling is fun in many ways, useful in certain aspects, but in the end should not be trusted as a guide or counselor for course decisions any more than one might trust a Ouija board. The bias is often missed, even though it is obvious once you know where to look. I have shown before that every poll is biased to some degree, and I have repeatedly made clear what you should expect from a valid poll:

• public access to internal data
• a consistent, publicly reported methodology
• weighting according to Census norms or reasonable objective standard
• archive data available for comparison to current polls

Alas, there are areas where even the best of opinion polls cannot meet the standard of true objectivity. The most common stumbling block comes when a poll weights its political party affiliation. Polling groups understand that a truly random system of contacting poll respondents will produce results which are, to some degree, at odds with the true opinion of the public as a whole. To correct for this, respondents are asked certain questions to determine key demographic data, and their responses are categorized according to those demographic keys and the overall results weighted so that the response pool model is in line with demographic norms. The demographic standard used is almost always based on the most recent US Census data, for reasons that this data is considered the most unbiased and reliable demographic data available. So, Census data is used for gender, race, age, education, employment, and economic strata norm determination. This makes a lot of sense, and I applaud the pollsters for that standard and diligence in its application. The same polls who are so careful to avoid bias in most demographic norms, however, get completely squirrelly when it comes to party identification.

Political party identification is a stronger factor in candidate support than any other cited demographic category. It is no shock, after all, that democrats overwhelmingly support the democrats' nominee, while republicans overwhelmingly support the republicans' nominee. The problem comes in, when the poll weights the response pool to match a desired party identification standard. I noticed four years ago when I looked at the opinion polls, that the polling groups changed their party identification weights, some every week! When I contacted the polls about this practice, most just ignored me, but a few did respond. I found their explanations troublesome, however. Pew, Gallup, and Rasmussen, for example, choose their party weighting by examining the self-reported party affiliation of respondents from a prior period, usually a month (the ones who change each week generally use a rolling average of respondents from the period reviewed). Unfortunately, the reasoning behind such weighting is directly contrary to the whole purpose of weighting in the first place. To see what I mean, let's apply that logic to other demographic groups.

- continued -

South Texas was hit hard by Hurricane Ike, and it is very unlikely that any polling group had much success calling anyone down here in the past week; those who had phone service restored had a lot to do, and no time for answering polls. But even as phone and power service was restored, this was done according to critical needs (like hospitals) and a plan to get the most service restored to the most people as quickly as possible, which in practical terms means that urban areas and recent construction would get priority. This is because of the concentration of population, and the relative ease of repairing lines which would be less likely to have serious damage to the lines and transformers (older neighborhoods tend to have trees and other growth obstructing lines, so that in a major storm older neighborhoods are much more likely to have substantial damage to the transmission lines). Consequently, for the next month, an overwhelming majority of poll respondents in South and East Texas will be urban areas and new neighborhoods, which would heavily skew the demographics of the polling, if those respondents were considered "normal" for the area's demographics. Or consider another example, where more mid-week polls are taken. People who work or who are taking care of children would be less likely to be available for polls, which would skew the apparent demographics towards the youngest and oldest voters, and those whose lifestyles matched the tactics used to reach respondents, like visiting malls or urban centers. Polling groups know these areas are heavily skewed in their demographic types, with heavy oversampling of certain groups, and so the weighting is adjusted to match Census norms, precisely because anything else would invalidate the poll results.

Those are just two obvious examples of why using poll results for one period to establish the demographic weights for another period would be clearly invalid; it's simply circular reasoning and any errors (and there are always errors to some degree) would be greatly magnified rather than corrected. What's worse, the only validity for any poll whatsoever is movement within a poll handled by consistent and transparent methods - if you shift party identification weights, you invalidate all conclusions. Anyone who has annoyed their lab professor understands that controls exist for a reason, and fiddling with the weights between polls is simply unscientific.

Some of the pollsters I have spoken with, argue that they shift their weights because no useful and objective source exists for party identification weighting. That, however, is not true. The exit polling from prior elections is a very valid source. It comes not from "adults", "registered voters", or even "likely voters", but from people who actually voted. And what's more, we can look at the last few elections to see if there is any substantive change. I looked at national elections for the past ten years, and discovered an interesting pattern:

In 2006, 38% of the voters were democrats, 36% were republicans, and 26% were independents;
In 2004, 37% were democrats, 37% were republicans, and 26% were independents;
In 2002, 39% were democrats, 38% were republicans, and 23% were independents;
In 2000, 39% were democrats, 35% were republicans, and 27% were independents; and
In 1998, 39% were democrats, 33% were republicans, and 28% were independents.

That sure looks like a consistent pattern to me. On average for the past ten years, democrats have averaged 38.4%, republicans 35.8%, and independents 26.0%. If we kick out top and bottom outliers, it becomes 38.7% democrats, 36.0% republicans, and 26.3% independents. Those numbers have a solid empirical history and a thoroughly objective source behind them, yet the polling groups do not use them. You might well wonder why.

Polling groups exist for one of three purposes. They either serve the needs of a client (private polling companies), they do academic research (which serves the faculty running the polls), or they are done for public release (including those by colleges like Marist and Quinnipiac). Each group has a very specific, and different, reason for existing. You may want to consider that both Barack Obama and John McCain have hired private polling firms, and consider why they each spend that money, as indeed every major political candidate does. And if political allegiance is so flighty, then why do so few of us know anyone who has supported one party, then another, in the same election? To read the reports from the polling groups, there are a lot of folks who loved Obama, then McCain, then Obama again, or vice versa, yet I have not met even one such person. Instead, I have seen democrats who stay democrats, I have seen republicans who remain republicans, and I have seen folks making up their mind as the campaign progresses. There are times where you might see a new jump in support, as previously undecided voters choose someone to support, but that happens for a reason - the "bounce" that supposedly comes just because a party has a convention, well, that's pretty bogus if you think it through. The democrats may have been more excited about Obama after the democratic convention, but I do not believe for a minute that it made many new democrats. I am quite sure that the GOP convention excited republicans, but again I have to say I do not recall hearing about a bunch of new republicans after the convention. So, when the polls increase democrats' weighting after the one convention and increase the republicans' weighting after the other, it's frankly dishonest because neither party saw much change in affiliation. What's more, does anyone really think that there were fewer republicans after the DNC, or fewer democrats after the RNC?

Please.

Party affiliation is not some ephemeral quality, which winks in or out of existence because of a speech by one candidate, the headlines of a single day, or the personal desire of a polling group to have a hot story for the newspapers and TV stations. Party affiliation is something developed over a period of time, and while it does change over time, it does so gradually, taking into account the evidence of history as well as events of the moment. There were tens of millions of republicans after Bob Dole's loss in 1996, just as there were tens of millions of democrats after Walter Mondale's thrashing by Reagan in 1984. Support for a candidate is a thing apart from party identification. When it does change, it is spurred by a significant event of meaning with regard to the values and ideals of the party This is not merely a belief, but the evidence of American electoral history and the employment of common sense.


TrackBack

TrackBack URL for this entry:
/cgi-bin/mt-tb.cgi/31797.

Comments (15)

Gee. I sure wish there was ... (Below threshold)

Gee. I sure wish there was an alternative universe where I'm actually a good singer and considerably younger than 53, George Bush is a great president, the American economy is strong, Wall Street is up by 370 points today, oil is spiking down by $15-25 a barrel today, and Iraq is an easy problem to solve where Shiites and Sunnis are the best of pals and perfectly willing to work together for the good of their country, and Al Qaeda is a volunteer social service organization of good deed doers. But alas, such things are only the things of Sci Fi.

Gee, I sure wish there were... (Below threshold)
P. Bunyan:

Gee, I sure wish there were an alternate universe where Paul Hooson wasn't an asshole, but alas...

Gee, I sure wish t... (Below threshold)
maggie:
Gee, I sure wish there were an alternate universe where Paul Hooson wasn't an asshole, but alas...

LOL!

Take it easy on Paul Hooson... (Below threshold)
Paul Duffau:

Take it easy on Paul Hooson, folks. He only comes here because it's lonely on his side of the fence and he's desparate for company.

It would be nice if he'd contribute something useful along the way - an insightful comment, a snarkless perspective or a bag of potato chips - but he is as he is and he offers up softballs most of the time.

Hey, now you're actually ad... (Below threshold)
mantis:

Hey, now you're actually addressing the topic you skirted around with your goofy look at the Gallup numbers: how to measure party affiliation. Your stated prescription still has problems, however:

weighting according to Census norms or reasonable objective standard

While the US Census collects registration data, it does not record political party affiliation (some states do, some states don't), so no luck there. Even if we could get data from them, guess what kind it would be? That's right, self-reported, just like the polls.

So what reasonable objective standard do you use? You say exit polls, but those have their own problems. First off, did you notice how wrong many of the exit polls have been in recent elections? If the exit polls don't match the results of the election, in terms of voting, why should pollsters trust those numbers two years later in terms of party identification? For instance, imagine you are a pollster, using previous exit polls to determine party affiliation weighting in the 2002 election. If you had averaged the exit poll numbers from the previous two elections, you would have dramatically underestimated Republican affiliation and overestimated independents, by about 4% apiece. Things change in two years; that's why pollsters don't usually use exit poll data from previous elections (except in comparison to the current numbers they are seeing). Plus, exit polling is self-reported data also.

A few other points...

What's more, does anyone really think that there were fewer republicans after the DNC, or fewer democrats after the RNC?

Yes and no. There may have been a good number of Republicans who were turned off by the McCain nomination before the convention, and when answering a pollster were more inclined to identify as independent. Then after the convention, they found that the Sarah Palin was warming their hearts to the party again, and would report a Republican affiliation. Since it's self-reporting, it doesn't really matter what party they are actually registered to, just what they tell the pollster (and this information gives the pollster a better idea of how that person will vote now than how they voted two years ago would).

The fact is, any method of arriving at party affiliation weighting is going to be flawed, short of current voter registration numbers that pollsters don't have access to. Actually that method would be flawed to, as registration does not necessarily equal turnout. Every method has flaws; that's why there's an MOE.

Party affiliation is something developed over a period of time, and while it does change over time, it does so gradually, taking into account the evidence of history as well as events of the moment.

And pollsters recognize this; the problem is with measurement. Using exit polling data as you propose is rejected by most pollsters because it is old data, because it is self-reported data, and because those that voted in the last election won't necessarily vote in this one. Imperfect as it may be, self-reported data from likely or registered voters right now has shown to be more reliable in determining weighting than historical data.

All of that said, your other expectations, of more transparency and access to methods and archival data, is spot on.

A question DJ - can we take... (Below threshold)
Paul Duffau:

A question DJ - can we take previous years polls right before the elections, re-weight with the exit data and then compare to the actual results? How close a correlation (within the MOE) would be achieved?

Long answers are okay but if I'm full of it, don't feel that you need to waste you time explaining - just tell it to me straight, it's cool.

It would be interesting to ... (Below threshold)
Hestrold:

It would be interesting to determine the political affiliation of the pollsters. Much like we know that most journalists, oh excuse me, 95% of journalists identify themselves as Democrats. Then they wonder why the rest of the country doesn't think like them. Now if pollsters were even 70% democrats, that would skew their polls. Human nature being what it is.

The new Gallup internals ar... (Below threshold)
arrowhead:

The new Gallup internals are out and McCain is running away with independents.

Dont trust the polls, this will be a landslide for McCain/Palin.

http://www.gallup.com/poll/108049/Candidate-Support-Political-Party-Ideology.aspx

And there you go. The only... (Below threshold)
mantis:

And there you go. The only group that McCain gained with was independents in the September 15 - 21 period. He stayed flat with conservative Republicans, and lost among every other group. Obama gained in all Democratic groups (and conservative Republicans, oddly), and lost among liberal/moderate Reps and independents.

I assume we'll get another post from DJ explaining how his previous two posts were based on his misreading the numbers.

Stop laughing.

Mantis,Democrats v... (Below threshold)
arrowhead:

Mantis,

Democrats vote for Democrats and Republicans vote for Republicans.

If I were Obama I would be worried that independents are strongly trending towards McCain.

Best wishes.

If I were Obama I would ... (Below threshold)
mantis:

If I were Obama I would be worried that independents are strongly trending towards McCain.

I'm not arguing about whom the poll is good for. Obviously the gains among independents for McCain is bad news for Obama. We've been talking about the numbers.

Actually mantis, we all kno... (Below threshold)
DJ Drummond:

Actually mantis, we all know very well what you are talking about: Denial. You have not hid it very well, is all.

As Arrowhead said, a pro-McCain shift among independents is a very bad sign for Obama, because the polls are weighting democrats more heavily than the historical record says they actually vote. While democrats will outnumber republicans at the polls, McCain's advantage among independents is statistically significant.

To use the popular phrase, a "game-changer".

The significance of subject... (Below threshold)
DJ Drummond:

The significance of subjective weighting causes published reports to be unreliable. As you know, mantis, I corrected teh date alignment on the party-to-overall matches, and demonstrated that the 'movement' was specious on that basis. Not just pro-Obama movement, but the supposed "bounce" for McCain. What has been happening is more subtle, but when measured on the longer term, more significant that is, any one week's numbers can be an outlier, but there are clear trends, which is what I was always pointing for readers to see.

Actually mantis, we all ... (Below threshold)
mantis:

Actually mantis, we all know very well what you are talking about: Denial. You have not hid it very well, is all.

Yeah, denying you the chance to dishonestly evaluate polls and receive nothing but praise for doing so. Sorry about that.

To use the popular phrase, a "game-changer".

Maybe so. I pay way more attention to state polls myself, and as such I'm not too worried. That national independent number is of concern, though.

The significance of subjective weighting causes published reports to be unreliable.

It's not subjective weighting, as I've explained now endlessly. You think pollsters should use exit polls from previous years to determine weighting, as if historical data were more useful than current data, and as if the pollsters hadn't considered that option. The difference, of course, is that they actually do statistical analysis to determine the best weighting scheme, whereas you just pull it out of your ass.

As you know, mantis, I corrected teh date alignment on the party-to-overall matches, and demonstrated that the 'movement' was specious on that basis.

You didn't correct anything, and you didn't recognize your original error. You still maintained that the numbers were flawed even though you didn't have the affiliation numbers for the tracking poll dates you said were evidence of the flaw. Now that the numbers have been posted you still ignore them, likely because the completely refute your conclusion (which was probably your starting point anyway).

any one week's numbers can be an outlier, but there are clear trends, which is what I was always pointing for readers to see.

Bullshit. Your contention was that pollsters were monkeying with the numbers to cover up gains for McCain (which turned out to have evaporated) based on no evidence whatsoever, a misreading of data, all to support your absurd contention that "trolls" were losing the election for Obama because he refused to rein them in. If your point were to emphasize the importance at looking at poll numbers long-term to identify trends and not get bogged down in week by week results, you would have gone about it much differently.

Actually mantis, we all ... (Below threshold)
mantis:

Actually mantis, we all know very well what you are talking about: Denial. You have not hid it very well, is all.

Yeah, denying you the chance to dishonestly evaluate polls and receive nothing but praise for doing so. Sorry about that.

To use the popular phrase, a "game-changer".

Maybe so. I pay way more attention to state polls myself, and as such I'm not too worried. That national independent number is of concern, though.

The significance of subjective weighting causes published reports to be unreliable.

It's not subjective weighting, as I've explained now endlessly. You think pollsters should use exit polls from previous years to determine weighting, as if historical data were more useful than current data, and as if the pollsters hadn't considered that option. The difference, of course, is that they actually do statistical analysis to determine the best weighting scheme, whereas you just pull it out of your ass.

As you know, mantis, I corrected teh date alignment on the party-to-overall matches, and demonstrated that the 'movement' was specious on that basis.

You didn't correct anything, and you didn't recognize your original error. You still maintained that the numbers were flawed even though you didn't have the affiliation numbers for the tracking poll dates you said were evidence of the flaw. Now that the numbers have been posted you still ignore them, likely because the completely refute your conclusion (which was probably your starting point anyway).

any one week's numbers can be an outlier, but there are clear trends, which is what I was always pointing for readers to see.

Bullshit. Your contention was that pollsters were monkeying with the numbers to cover up gains for McCain (which turned out to have evaporated) based on no evidence whatsoever, a misreading of data, all to support your absurd contention that "trolls" were losing the election for Obama because he refused to rein them in. If your point were to emphasize the importance at looking at poll numbers long-term to identify trends and not get bogged down in week by week results, you would have gone about it much differently.

Oh, by the way, read this.




Advertisements









rightads.gif

beltwaybloggers.gif

insiderslogo.jpg

mba_blue.gif

Follow Wizbang

Follow Wizbang on FacebookFollow Wizbang on TwitterSubscribe to Wizbang feedWizbang Mobile

Contact

Send e-mail tips to us:

tips@wizbangblog.com

Fresh Links

Credits

Section Editor: Maggie Whitton

Editors: Jay Tea, Lorie Byrd, Kim Priestap, DJ Drummond, Michael Laprarie, Baron Von Ottomatic, Shawn Mallow, Rick, Dan Karipides, Michael Avitablile, Charlie Quidnunc, Steve Schippert

Emeritus: Paul, Mary Katherine Ham, Jim Addison, Alexander K. McClure, Cassy Fiano, Bill Jempty, John Stansbury, Rob Port

In Memorium: HughS

All original content copyright © 2003-2010 by Wizbang®, LLC. All rights reserved. Wizbang® is a registered service mark.

Powered by Movable Type Pro 4.361

Hosting by ServInt

Ratings on this site are powered by the Ajax Ratings Pro plugin for Movable Type.

Search on this site is powered by the FastSearch plugin for Movable Type.

Blogrolls on this site are powered by the MT-Blogroll.

Temporary site design is based on Cutline and Cutline for MT. Graphics by Apothegm Designs.

Author Login



Terms Of Service

DCMA Compliance Notice

Privacy Policy