Historically, Mystery Pollster told his audience that he gave the 30% number to Freeman. He gave it to him off the top of his head based on what he thought was the historical number.
Then Mystery Pollster reversed himself after consulting several authorities and doing "research" and was told that this year it was more predicted to be between 50-80%. This made the MOE's be 3.5% all the way up to 5% or more.
I gave him plenty of grief on his website for a margin of error of 3.5% because it would not be useful to predict any of the swing states and the networks weren't going to pay the kind of money they pay for something no more accurate than throwing darts at a dart board.
Here's what I told him about ten days ago with a few updated edits:
...........
I'm just a lawyer but as I have told several mathematicians, this is important so I am trying very hard to learn about the holy grail of "design effect." The first thing I have learned is that some mathmeticians think this notion of "design effect" is completely bogus as it applies to exit polling.
Others think it is a number pulled out of someone's butt to make exit polling match actual polls.
I initially bought the concept but now I am beginning to have second doubts.
I understand the concept of clustering. If you put 10 clusters of 50 red marbles in a jar and 10 clusters of 50 blue marbles in a jar and don't mix up the jar, obviously if you take 50 marbles out at random you run the risk of getting 30 or 35 reds or 30 or 35 blues instead of something close to 25 each. So I understand if you have clustering, you have to account for this effect.
The argument goes that you have clustering (democrats and republicans live in geographic clusters)in exit polling so you must account for this effect. But the effect of clustering only occurs with small sampling. If you sample 1000 clustered jars, the 35 reds you get in one jar will be offset by 35 blues in another and the effect at some point will cancel itself out.
The higher the population, the more samples will be required; thus as our population has increased, the design effect should be headed towards extinction. Surprisingly though, the "experts" say the design effect is heading in the other direction. According to your recent post, pollsters used to account for a 30% design defect in 1996 (I think) and now it has increased to between 50-80%.
My second concern is whether we really are in fact polling in "clustered" places. More and more I see graphs of a purple America, not a blue and red one. Yes there are a few truly red areas and a few truly blue areas, but I have a hard time believing that the average polling place is much greater than a 70/30 split. Probably most are less than that. This would certainly reduce any design effect.
Third. If there is a design effect due to the clustering of exit polling, it should be evident when compared to pre-election polling which is supposed to be random. Random sampling and clustered sampling are not supposed to produce the same results. But in a study done on Democratic Underground by a mathemetician who calls himself Truth is All he compared the very last pre-election polls with the latest "non-compromised" post election polls and instead of the numbers being different as one would expect if there were a design defect, they were almost identical. He is one of the mathematicians who thinks the design effect concept is bogus to begin with. He was not attempting to show the design effect was a bogus concept by the study. I pointed it out to him. He thinks the design effect is so bogus he doesn't even want to engage in the argument. But his results should be evidence that the claimed design effect doesn't exist or has cancelled itself out due to the volume of samples.
My fourth problem is a business one. If you include the design defect in your calculations it means that margins of error are easily 3.5% in the best of situations. I can not imagine networks paying the millions of dollars they pay for election night to only be within a 3.5% margin of error when the polls close. You would not be able to call any of the swing states on anything like the timetable the networks want. I don't buy the argument that they pay all this money for other data. Sorry.
Fifth. Another mathematician on Democratic Underground, (who believes in the concept of clustering) has reworked the numbers using the highest numbers for the design defect you have given. His opinion of the data is opposite yours. His name is macdonald. But he has no idea what the real numbers for design effect should be; he's just using the latest ones you have.
Sixth: Why were exit polls so accurate in the past but not now?
Seventh: Why are they still accurate in Europe? Presumably they have political clusters there and they can get it right.
Eighth: Could it be that whatever factor you use for clustering is only as good as the honesty of the last election?
Ninth: Could it be that computer fraud has caused these numbers to inflate?
Tenth: Another mathematican on Democratic Undergrond has shown that the exit polls of those states with paper ballots were far closer than those that used paperless systems. His name is jwmealy. He suspects computer fraud.
So please forgive me if some of us don't buy the cluster argument yet.
You can check these other mathemeticans out at:
http://www.democraticunderground.com/discuss/duboard.php?az=show_topic&forum=203&topic_id=132803I do know of one way to test the clustering phenomonon I think. Randomly poll about 40,000 Americans now and see how they voted. That should give us a pretty accurate idea about a lot of things, including the right numbers to use for the clustering effect.
Posted by: davidgmills | December 14, 2004 09:14 PM
......
As you know, I have proposed re-exit polling Ohio because I believe you are right. I have challenged your main nemisis on Mystery Pollster, Rick Brady, to come up with the number of people he would consider necessary to have a 1% MOE has said it would take a re-poll of 16,000 voters.
I say re-poll Ohio now and give Conyers something unquestionably concrete (even to the staunch opposition) to work with.