Nate has conducted some fascinating analyses and provided a terribly important service to us poll watchers. I've provided snippets. Check out the whole thing at 528 if you're inclined.
Strategic Vision Polls Exhibit Unusual Patterns, Possibly Indicating Fraud
by Nate Silver @ 9:04 AM
Share This ContentOne of the things I learned while exploring the statistical proprieties of the Iranian election, the results of which were probably forged, is that human beings are really bad at randomization. Tell a human to come up with a set of random numbers, and they will be surprisingly inept at trying to do so. Most humans, for instance, when asked to flip an imaginary coin and record the results, will succumb to the Gambler's Fallacy and be more likely to record a toss of 'tails' if the last couple of tosses had been heads, or vice versa. This feels right to most of us -- but it isn't. We're actually introducing patterns into what is supposed to be random noise.
Sometimes, as is the case with certain applications of Benford's Law, this characteristic can be used as a fraud-detection mechanism. If, for example, one of your less-trustworthy employees is submitting a series of receipts, and an unusually high number end with the trailing digit '7' ($27, $107, $297, etc.), there is a decent chance that he is falsifying his expenses. The IRS uses techniques like this to detect tax fraud.
Yesterday, I posed several pointed questions to David E. Johnson, the founder of Strategic Vision, LLC, an Atlanta-based PR firm which also occasionally releases political polls. One of the questions, in light of Strategic Vision LLC's repeated failure to disclose even basic details about its polling methodology, is whether the firm is in fact conducting polling at all, or rather, is creating fake but plausible-looking results in order to increase traffic and attention to its core business as a PR and literary firm.
I posed that question largely as a hypothetical yesterday. But today, I pose it much more literally. Certain statistical properties of the results reported by Strategic Vision, LLC suggest, perhaps strongly, the possibility of fraud, although they certainly do not prove it and further investigation will be required.
<SNIP>
I haven't really seen anyone approach polling data like this before, and I certainly haven't done so myself. So, we cannot rule out the possibility that there is some mathematical rationale for this that I haven't thought of. But it looks really, really bad. There is a substantial possibility -- far from a certainty -- that much of Strategic Vision's polling over the past several years has been forged.
I recognize the gravity of this claim. I've accused pollsters -- deservedly I think in most cases -- of all and sundry types of incompetence and bias. But that is all garden-variety stuff, as compared against the possibility that a prominent polling firm is making up numbers whole cloth.
<SNIP>
http://www.fivethirtyeight.com/2009/09/strategic-vision-polls-exhibit-unusual.html David E. Johnson of Strategic Vision (not to be confused w former Florida GOP executive Director David Johnson) categorically denied the allegation and suggested legal action may be looming. Here's Johnson's statement:
<SNIP>
Secondly in regards to Nate Silver's statements, we categorically deny them and will refute them. We have a call into our attorney on this and fully intend to take action that will vindicate us. I wish Nate had contacted me directly yesterday when he began this tirade, I could have answered his questions fully to his satisfaction prior to damage being done to our reputation. Now that he has made these accusations and posted them online, I must and will defend our company's reputation through all legal avenues available. The reason that we are going the legal route is he has attempted to do severe damage to our reputation and what is he going to do when we disprove him just say I am sorry. That isn't enough at this point.
http://blogs.tampabay.com/buzz/2009/09/nate-silver-pollster-may-be-fraud.html