Any poll experts out there to answer a question?

[WilliamPitt]

Zogby has Kerry and Dean three points apart. ARG has Kerry almost twenty points ahead. The local polls here in Boston - Suffolk, Boston Globe, WBZ - have different numbers, too. What explains the vast differences in the numbers? I can't recall seeing so many polls on one topic being so wildly divergent. Help?

[BootinUp]

1. non-expert opinion

Very Fluid.

There is a large percentage of voters that can't make up their stupid minds.

[gWbush is Mabus]

2. stats101

sample size
sample composition (random or biased)
wording of the questions asked

[WilliamPitt]

3. Ah.

Interesting. Wish we could see the questions.

[HFishbine]

12. Some insight

Here are the questions for the Suffolk poll: http://www.suffolk.edu/suprc/pres/jan26_04/pdf/MARGINALS.pdf

Something I heard explained about tracking polls, that made sense to me, is that they typically use a small sample each day, 200 voters, for example. Which has a very high margin of error in NH (about 10%). But the pollsters will multiple 200 by the three days (600) and say that a three day average has a MOE of only 5% because it represents 600 voters. The pollster who was explaining this insisted this is statistically invalid because it actually compounds a high MOE instead of mitigating it.

[spooky3]

14. good point

[hlthe2b]

21. That is correct... Three independent samples collected at different times

can not be reliably combined and the margin of error for each is compounded. While this is a sampling tool that can be used for "fixed opinion" surveys (e.g., National Health Information Surveys to assess health behaviors)in a population sample that was pre-selected, it is horrendously unreliable for assessing opinion when outside factors are still in play (i.e., campaigning is ongoing).

[HFishbine]

27. Yes

That furthers the explaination I heard which was by that grumpy bearded pollster, I think his first name is Pat...

[ozone_man]

10. Whether push polling is used.

E.g., I heard in Iowa that some polling preceded the candidate preference question with a question as to whether Dean was electable.

[creativelcro]

17. All true. Zogby was on target in Iowa, but there was less divergence

there. Zogby says he's using "random" samples. Of course, truly random samples don't exist in this field. If somebody is not home that person is not sampled, even if the random selection picked that person. And the reason why people may not be home may have a pattern. Just an example. The wording of the question is CRUCIAL. Not showing the questions makes the entire biz impossible to evaluate. The sample size should not be a huge problem. Zogby reports it, as do the others. I don't think the polls reported have a theortical MOE of +/- 10 points.

Given the unexplained variance, basically the polls are more useless than ever, right now.

[spooky3]

18. link?

My recollection was that none of the polls predicted the %s even within large margins of error. Zogby might have had the order of finish correct, but I am not even sure about that.

Zogby's main claim to fame was predicting the 2000 presidential election results.

[creativelcro]

22. Zogby mainly "predicted" that gephardt would be last.

the rest of the numbers were within the margin of error. So, the fact that, numerically, Kerry came out ahead in the poll probably was just chance...
www.zogby.com

Final Zogby Iowa Poll

Mon Jan 19th, 2004 at 12:30:35 GMT

If the caucus were held today, for whom would you vote?
Kerry: 25%
Dean: 22%
Edwards: 21%
Gephardt: 18%

From the same list, who would be your second choice?

Kerry: 25%
Edwards: 21%
Gephardt: 15%
Dean: 13%

If you had to choose today, which candidate are you leaning toward? (Asked only of those who responded "Not sure" to the first question.)

Kerry: 20%
Edwards: 14%
Dean: 12%
Gephardt: 8%

Overall Opinion - Favorable/Unfavorable

Kerry: 82% to 12%
Edwards: 75% to 10%
Gephardt: 72% to 20%
Dean: 65% to 32%

Spread

Kerry: +70%
Edwards: +65%
Gephardt: +52%
Dean: +33%

[spooky3]

29. Thanks, but the actual results are NOT within the MOE

if my recollection is accurate that that MOE was about 4.5. Even if I round up to 5, Zogby is wrong, except for Dean, and that was just barely inside the MOE.

Kerry--predicted 25, add MOE = 30, actual 38
Dean--predicted 22, subtract MOE = 17, actual 18
Edwards--predicted 21, add MOE = 26, actual 32
Gephardt--predicted 18, subtract MOE = 13, actual 11

and that is my point. The pollsters generally have huge MOEs but still are wrong.

[RUMMYisFROSTED]

33. You're not taking into account the delegate swapping.

The initial head count for each candidate was within the MoE. After the delegate swapping, Kerry and Edwards added to their totals.

[spooky3]

34. link to support your assertion about the initial head count?

You're definitely right that the caucus is even harder to predict than simple voting. But I don't think that had much of an impact but would be happy to see evidence to the contrary.

So maybe all the pollsters will be closer on NH. We'll see.

[RUMMYisFROSTED]

44. Here's one:

Kerry Leads Early Caucuses Poll

An entrance poll at the Iowa Democrat caucuses shows Sen. John Kerry leading, with the support of 29 percent, Fox News Channel reported tonight.

Sen. John Edwards had 22 percent, Howard Dean 21 percent and Rep. Dick Gephardt 16 percent, according to the early poll.




https://www.newsmax.com/archives/ic/2004/1/19/204356.shtml

Sorry for the source, it was the first one that popped up.

[creativelcro]

38. Yes, I agree... I rechecked the number and that is correct

[kentuck]

24. I think they need psych101 more than they need stats 101...

They are playing with people's minds and no one knows how they might react.

[AP]

28. because they make different presumptions, too, and give different demos

different weight, based on history and trends.

For example SC Arg is presuming very low black turnout and are weighting it at 39% (I've heard). Most other weight it at 45-50. Some people think turnout will be over 50%.

Things like this wouldn't matter if the black vote were evenly distributed. If black voters' support isn't evenly distributed, a difference like this could skew polls with different assumptions greatly.

In NH I bet one big difference across polls is the weight given independent voter turnout.

[Guaranteed]

42. To begin with, Zogby is polling "likely voters"

while the other polls are only polling registered Democrats.

I'm sure there are other factors, though, too.

[Wickerman]

4. Lots of factors

A quick one - I just heard on TV (so it must be true) the Zogby "forces" takers into a candidate - so they will get different and quite possibly skewed data. Plus, NH is a largely independent state - so if you are polling for likely voters you can get a weird mix. If you poll only for registered Dems you would get an entirely different set of results.
Without seeing the polls and the questions those are just a few things that go into making this as large a crapshoot as Iowa.

[spooky3]

5. here are some factors that influence poll accuracy

1) Is the reality stable or are people changing their minds? Even the best poll can sample only what people are saying right now.
2) Do you have a random sample of the population, or has your sampling procedure (and any correction that you may do of it) led to your over-sampling people of certain persuasions, and an underrepresentation of others? No poll gets close to a random sample, in contrast to other types of studies. So each of these polls is probably getting samples with different characteristics, rather than a truly representative group of people from the population.
3) How big is sample size? If you only ask 300 people, even if truly random, from a population of thousands or millions, your poll will be less accurate than if you could sample 1000.
4) What is your response rate? If only 1 in 10 people that you try to poll are able and willing to talk to you, chances are that you do not have a representative sample of the population. That's about what the response rate is on these polls.
5) How high is the quality of the questions? If you ask the questions in such a way that the respondent is encouraged to respond one way or another (even subtly) or encourage respondents to make decisions about things they don't know much about, you are not going to have accurate results.

[Monte Carlo]

6. I know something about stats, and...

... in every poll you've ever heard, there's a little tidbit they never mention.

Say there's a poll like NH ARG, and they give a margin of error of +/- 5%. What they don't tell you is that they are never 100% confident that their results lie within +/- 5%; that's a statistical impossibility.

Usually, polls will have a confidence level of around 95%. That is, for example, these NH ARG poll results have a 95% chance of being within +/- 5% of the true support numbers. They could very well be 10%, 15%, or even more off from the true numbers. And if one poll can be inaccurate, then they can all be inaccurate.

Does that help?

[WilliamPitt]

8. It does

Thanks.

[HereSince1628]

13. Also, the MOE for the poll isn't the CI for each estimate.

The confidence intervals for the estimates get larger (the estimates are more uncertain) as the percentages get smaller...since the percentages are mostly under 20 there is a lot of noise in play.

[BootinUp]

36. I think this is right

My own knowledge of statistics is based on years of experience in quality control not politics, however your noise comment agrees with my observations. The pollsters seem to want us to believe that as the number of candidates in the poll increases there is no effect on MOE or confidence. Preposterous.

[DjTj]

30. Isn't a MOE usually just 1 standard deviation?

...which would only be 68%...

[spooky3]

31. no, MOE has a specific calculation. It is primarily influenced by sample

size. You may be thinking of the confidence interval.

If you google with these words

margin of error american statistical

The first item gives you a .pdf file with a description of MOE from the Am. Statistical Assn.

[onehandle]

7. Another non-expert opinion

Due to caller-id, answering machines, cell phone use, and no call lists the pool of who will actually respond is getting smaller.

Imagine who would actually respond to a pollster? Every group an organization gathers would be so damn different. I think the poll companies won't admit that polling just doesn't work anymore.

I haven't talked to a salesperson or pollster in years.

[mdmc]

9. I hear you knocking but you can't come in

Consider who is being asked, when, and how. Poor people are never considered, for instance, because they are so hard to reach (no phone, working all the time).

[Raven]

11. I was called on Sunday by Gallup.

The questions were actually difficult and somewhat confusing...even for someone like me who has made up their mind.

[MaineDem]

15. You're the first I've heard of someone being polled

Finally! :)

[redqueen]

40. shock shock horror horror

shock shock horror


Who could have thought?

[flpoljunkie]

16. Zogby's caveat today...he included "leaners"... pushed to pick favorite

*In today's release, Leaners among the Undecideds have been factored in. Without Leaners, the percentage of Undecideds is 13%.

http://zogby.com/news/ReadNews.dbm?ID=793

[isbister]

35. Included leaners and ...

I do not believe they have changed the margin of error... seems to me that the +/- 4.1 has been up there since before the leaners were included. I'm no expert but including leaners has got to increase the error rate.

[robsul82]

19. I don't know, but Zogby better be right.

Sons o' bitches.

Later.

RJS

[markus]

20. Most polls are conducted over multiple days

This tends to even out irregularties in sample based on who's home when, etc.

It's still risky (but highly entertaining) to speculate on one day movements in tracking polls, but you're really getting into the steamy entrails of polling at that point.

At that point, you have to rely on the pollster's expertise. It's like the time I got a hairline fracture, and the ER X-ray guy couldn't find it. Went to my bone doctor the next day. He took one look at the X-ray and said, "oh, there is." You have to rely on the professional gut and skill of the person who best knows the data source, methods, etc.

Our own speculations (and that of the Gibbering Pundits) is just that.

My own experience of over a decade ago (when I occasionally had the priviledge of being on these meetings or calls is), one day's tracking is interesting; two days in the same direction is significant; three days, call your bookie).


On Edit: There's a reason the ABC Note refers to tracking polls in general (and Zogby in particular) as "crack for the week minded". Loads of fun, highly addictive, and very bad for you in the long run.

[kentuck]

23. The pollsters were all wrong in Iowa, except Zogby....

Most of them were picking Dean/Gephardt and it came out Kerry/Edwards. I think most of the professional pollsters are now going with the wave that came out of Iowa. I think may understand that there are unknown factors involved, psychology and strategic thinking of voters, that have not been there before.

That said, I think Zogby is probably closer to correct this time also. I think the person that might throw all the pollsters for a loop is John Edwards. Once people go behind the curtain to vote, many see something in Edwards that they find very comfortable and confident with, and I think that is a factor in NH that they are all overlooking.

[spooky3]

32. Zogby was right only in picking Kerry first and Gephardt last

but was wrong about Dean and Edwards' order of finish. All of the candidates' actual percentage were much higher or lower than he predicted, even taking his MOE into account, except for Dean's, which barely made it inside the MOE. See above posts from creativelcro and me.

To be a good result, all of the candidates' percents should be within the margin of error, and the closer they are to the predicted %, the better the poll.

[creativelcro]

39. He picked Kerry first by chance though...

The actual MOEs did not allow Zogby to pick, technically... I mean, technically, the poll results were compatible with all possible orders because they were all within a 9% range.

[spooky3]

41. good point

[Skwmom]

25. I believe moderateindependent.com

discussed the polls and how they are gamed in NH. Plus, have you been reading up on some of the pollsters. They do the bidding of the republican party. By putting out polls they are hoping to make a self fulfilling prophecy.

[JI7]

26. they use different methods

the factors they take in to adjust the numbers to try to make them more realistic are different at times . some might adjust numbers based on history. some might do it based on what things may look like on the campaigns.

[KaraokeKarlton]

37. None of the NH polls should be taken seriously

Because they don't do a realistic sampling of the voters there. About 40% of NH voters are registered Independents. Only 26% are registered Democrats. Independents can vote in the primary and they often do. The whole reason that NH is so unpredictable in elections is because the Independents always decide who is going to win in NH. If pollsters were wise, they would do a separate poll of only Independents in NH to see who they support and a poll of just the Democrats and then merge the totals to get an accurate clue as to how NH is going to go. They don't do that, though, so truthfully, none of the pollsters have any idea who is the likely winner. Some will argue that the ARG poll does poll 30% Independents. However, they only poll Independents who always vote for Democrats in regular elections. Those aren't the only Independents who will be voting. Furthermore, if you take 70% of your sample from the voting minority and only 30% from the voting majority you simply are NOT going to have an accurate sample, period.

NH Independents like Howard Dean and Wes Clark. A few like Lieberman. NH is a huge gun state and big on balanced budgets and fiscal responsibility. So, most of the Independents are a wee bit on the more conservative side. Not right wing conservative, but conservative in the New England tradition.

[Snivi Yllom]

43. zogby seems to have pushed undecided voters

and they seem to have gone to Dean.....

[Walt Starr]

45. New Hampshire has been notoriously dificult to predict with polls

That said, I expect Kerry to win with double digits and predict that Lieberman will pull out a squeaker over Dean to take second. Edwards will be fourth and Clark will be a disappointing fifth.