My response to the latest Mystery Pollster (Jan. 8) blog

TruthIsAll

(1000+ posts) Send PM | Profile | Ignore

Mon Jan-10-05 08:39 PM
Original message

My response to the latest Mystery Pollster (Jan. 8) blog

Edited on Mon Jan-10-05 09:17 PM by TruthIsAll

Just browsing his site. It seems that the "smoking gun" has left some tracks. I offer my 2c here, for all you DUers who have been reading my stuff these past 6 months.

I'm sure many of you have heard of Sandy Koufax, the great Dodger pitcher. For four years running (1963-1966) he was by far the best pitcher in baseball. Sandy came over the top, the same way, curve or fastball. His delivery was straight and true. Perfection. A true scientist on the mound. No tricks. Just a hard fast ball and a wicked curve. No one could touch him.

Sandy was to the science of pitching what Ted Williams was to the science of hitting. Both were true masters who continually refined their craft. Like Williams, Sandy had two supreme gifts: physical and mental. And he was committed to the truth in everything he ever did in life, before and after he quit baseball at the age of 30 in 1966.

I know. I saw them both play.

Having said that, congratulations to the Mystery Pollster. He has just won the Sandy Koufax Award.

http://www.mysterypollster.com/

snip

MP:
The point I wanted to make then was the exit polls were obviously wrong in some ways, not so wrong in others. Everyone, even Mitofsky, concedes that the just-before-poll closing exit polls had an average "error" (or, to some, a "discrepancy") of roughly 2% in Kerry's favor compared to the actual count.

Where the exit polls were right - or at least, not quite "wrong:" The errors were too small to achieve statistical significance in all but a handful of statewide polls. They were not large enough to give Kerry a lead beyond sampling error in any states that he ultimately lost, and not large enough to result in any wrong calls on election night.

TIA:
It does not matter where the discrepancies occurred. They added to Bush�s popular vote, if not his electoral vote. But consider that Ohio and Florida were among the sixteen (16) states out of the forty-one (41) which deviated to Bush beyond the Margin of Error, based on exit polling sample size. Not one of the ten (10) states which deviated to Kerry fell beyond the MOE.

I say that is VERY SIGNIFICANT.

That is Very Significant. I�m sure you are already familiar with the odds: It's 1 in 13.5 trillion that this occurence could have been be due to chance.

MP:
.

Where the exit polls were obviously wrong: As the Washington Post's Richard Morin put it in November, the errors were "just enough to create an entirely wrong impression about the direction of the race in a number of key states and nationally." And Kaus is right -- it wasn't just bloggers, but sophisticated journalists and political insiders who reached the wrong conclusion looking at those numbers on Election Night.

TIA:
The wrong conclusion? You mean the journalists should have believed the votes and not the exit polls? Why should they not believe their lying eyes?

MP:
Of course, supporting the official network projections is only one mission, and arguably the least important. The exit poll subscribers also pay to get (a) some early indication of the outcome on Election Day so they can plan their coverage and (b) data to support analytical stories written on Election Night that explain the outcome and characterize the race among demographic subgroups. Here the exit polls obviously failed. News organizations planned coverage on the assumption that Kerry would win. Some stories based on the early evening cross-tabs apparently had to be rewritten. As John Ellis -- a former analyst for both NBC and Fox News -- wrote on his blog shortly after the election:

The lost productivity at places like The New York Times and The Washington Post, where literally hundreds of reporters and editors spent the equivalent of an 8-hour work day writing and preparing fiction...all of the consumers of this content have to be asking themselves: "why in the world do we pay for this?"

TIA:
John Ellis is Bush�s cousin. If you were not aware of that, you should be. He was the first to call Florida for Bush in 2000, after 16,022 votes were mysteriously dropped from Gore�s total in Volusia County and spread around to various third party candidates (the Socialist candidate got 10,000). Dan Rather called it a oomputer �glich� at the time. By the time the error was �corrected:, it was too late; the media had already declared Bush the winner � all because of a �glich�, thanks to your very own John Ellis.

MP:
So who is to blame for that wrong impression? That is the central argument between Mitofsky and Kaus and others. Was it Edison/Mitofsky for how it managed and disseminated the results? Should the networks have spent more to assure better interviewing and coverage? Were they both wrong to resist disclosure of basic methodological details that might have helped reporters and editors and even bloggers better understand the limits of exit polls? Should those editors, reporters, and bloggers have known better? I tend to exempt the consumers, but otherwise, I find it difficult to place all the blame in one place, especially given how little we really know about what went wrong and why.

TIA:
You ask who is to blame for giving the �wrong impression�. Since when do exit poll numbers give the �wrong impression�? Perhaps the unseemly early tabulation of votes for Bush in the East was giving the �wrong impression� and the exit polls were telling us the truth. You are being quite unscientific in your failure to consider that scenario.

MP:
I am willing to cut Mitofsky some slack -- at least until I know more -- about the nuts and bolts of why the exit polls were off. However, I tend to agree with his critics in one respect: The lack of transparency about basic methodology, the instinct to deny obvious problems and then blame the bloggers and his habit of lashing out in anger at criticism are at odds with someone of Mitofsky's well deserved reputation and stature.
TIA: At least you agree on that one.

January 04, 2005
THOSE LEAKED PDFS

About those newly leaked exit poll numbers. Early this morning, Mickey Kaus linked to my earlier post the recently leaked exit poll documents and declared them a "smoking gun." When I first scanned the documents yesterday, I did not see the same bombshell. I still don't but, let's take a closer look.

First, let me clarify the source: Scoop, a left-of-center website in New Zealand that has been following the various vote fraud conspiracy theories, somehow obtained copies of the pdf reports that NEP provided for its national and regional exit polls. Since Scoop provided no state level data, it is likely that their source was a newspaper subscriber, not one of the NEP network partners. By the way, the "scoop" on the scoop numbers came from on New Year's eve from the Blue Lemur, another "progressive" site.

TIA:
You say that SCOOP has been following the various fraud conspiracy theories. Well, MP, your bias is showing on that one. The statement speaks for itself. You call �Lemur� another progressive site, left-of-center site. What does this have to do with the data? The data does not have a political philosophy; it speaks for itself.
How should we label your site: Right-wing, conservative, or neo-conservative?

MP:
The documents do tend to confirm what had been reported elsewhere (links to documents are in the bullet points that follow; all times are presumed to be EST):

� On Election Day, the "national" exit poll had Kerry ahead by three points (51% to 48%) at 3:59 PM and by the same margin (51% to 48%) at 7:33 PM when the polls were closing on the east coast. By 1:33 PM the following day, the completed, weighted-to-match-the-vote exit poll showed Bush leading (51% to 48%). These numbers had been previously reported by the Washington Post's Richard Morin and Steve Coll on November 3.

� The early samples included too many women: The percentage female fell from 58% at 3:59 PM to 54% at 7:33 PM, but this change alone did not alter the overall candidate standings (as the Simon/Baiman paper argues). By the next day, the sample was still 54% female, but the results among men and women were very different - Bush was 4 percentage points higher among men, 3 points higher among women.

� All of the three releases are marked as "weighted," but keep in mind: The first two releases were weighted only to bring their geographical distribution into line with hard counts of actual turnout. The last release would have been weighted so that it matched the official count (something I explained here).

TIA:
So we should believe that the last poll was weighted to match the official count? You mean �manipulated�, don�t you?

MP:
� Keep in mind that the 7:33 PM sample from election night was incomplete. It had 11,027 interviews, but the next day NEP reported 13,660. The missing 2,633 interviews, presumably coming mostly from states in the Midwest and West, amounted to 19% of the complete sample (The Simon/Baiman paper includes what appears to be a later and more complete national survey report - more on that tomorrow).
The margin of error provided by NEP for the national exit poll was +/-1%. Thus, Kerry's early lead and the overall differences between the 7:33 p.m. and 11/3 numbers were statistically significant. That the errors in the national poll were statistically significant, while similarly sized errors in state exit polls were not, owes to the much larger sample size of the national survey.

TIA:
Yes, the later survey report on the Washington Post web site was that of 13,047 RANDOMLY-SELECTED voters. That gives a 1.0% MOE. I guess you will finally stipulate that the MOE is 1.0%, at least for the National Poll. And that the Kerry numbers were statistically significant. Well, we are making priogress. But I�m sure that you will not leave it at that.

MP:
So is there anything truly new in these documents?

Perhaps not. At least not to me, and hopefully not to MP's faithful daily readers. However, to political sophisticates like Kaus who do not share our odd er...enthusiasm for the exit poll controversy, the official documents have more power than a few lines buried in an online chat. The last time most political junkies checked in on this story, it was a few days after the election and the "blame the bloggers" meme was in full force. You remember: The problem was not the exit polls or the way they were handled by the networks, but the foolish bloggers who blabbed about "early numbers" they did not understand. Well, these documents confirm something loyal MP readers have long known - the just before poll-closing numbers had the same errors.

TIA:
Of course you won't leave it at that. You see nothing new or significant? Hmm� I would think that it would raise some curious eyebrows. Foolish Bloggers? Is that how you would characterize Baiman or Freeman or Simon or others whose analyses are based on that WP 13,047 sample?

I seem to recall your argument that Exit Polls are not randomly-selected samples. And that the MOE is much higher than for pre-election polls due to �cluster�.

How do you explain the fact that Mitofsky-Edison say nothing about �cluster� in their notes on the WP/NEP 13,047 sample? And how do you explain the fact that they call the NEP a randomly-selected sample with a 1.0% Margin of Error?

MP:
Though my perspective is different, I am struck by how little guidance these cross-tabs provide about statistical significance. A newspaper subscriber like Richard Morin would have to do what I did above: Look up the sampling error provided by NEP in a separate table and apply it separately to each number. I cannot imagine that many editors or political writers went to that much trouble.
By comparison, I am told that NEP provided the network "decision desks" with printouts (or computer screens) that provided the exact confidence level for every vote estimate. That is, the estimate for each state included a percentage indicating the statistical certainty of the leading candidate winning the state. Networks would consider calling a state for a candidate only when that percentage went over 99.5% certainty. I had assumed the national poll reports provided to newspapers included similar reports of statistical significance. That they did not may explain why the newspapers that subscribed to the exit polls have been more willing to complain about the exit polls in public.

TIA:
It's NOT a question of calling the WINNER of a state. It IS a question of how the DEVIATIONS from the exit polls could be so dramatic and so far BEYOND the margin of error. THAT is the issue which you conveniently sidestep.

MP:
The continuing stonewall of secrecy that the networks have erected around the exit polls does not help. It is that secrecy, as much as anything else, that continues to fuel the more bizarre conspiracy theories floating around the blogosphere. I remain a skeptic of widespread fraud, but I cannot understand the continuing secrecy: Why did these documents have to be leaked by a left-wing web site in New Zealand? Why did NEP not release them in early November? Why did it take until late December for NEP to make the basic methodology statements the networks had on Election Day available online? And why so much reluctance to discuss, openly, what went wrong and why?

TIA:
You refer once again to "left-wing" web sites. How right-wing of you. And your choice of adjectives is very revealing: Bizarre. As if any questioning of the validity of the votes based on exit polls is a bizarre enterprise. Tell that to the Ukrainians.

MP:
A bit more transparency from news organizations that trumpet our "right to know" would certainly help.

TIA:
It sure would.