|
Editing period expired--school nurse called, had to talk to wife, etc., etc.
--------
I've seen long-settled "science" overtuned by B, a grad student, in front of his lab PI and mentor. I've told my advisor point blank that his article, already at the presses, was untenable. In a seminar we had the professor essentially chuck the syllabus in week 3 because his previous topic was too indefensible. I've had one committee member tell me that my project was simply unworkable until I asked him to explain a series of sentences in his native language.
Take the first example. B had to re-present this foundational paper, one that was quoted in standard undergrad texts as fact and confirmed by a dozen researchers in the following 6-7 years and assumed true for scores of other papers. He had said he had a funny feeling, and his dissertation mentor, one of the paper's authors, told him he needed practice presenting and told him to trust analysis and not a "funny feeling." B had integrity. The first 30 minutes of B's presentation was looking at the math behind the rather clever statistical tools the authors used, and even the PI was chafing. Until B got to the end, where he made it clear that the assumption built into the math was that whatever you think the degrees of freedom are they're one less. One is assumed by default. It had to be the PI's assumption as well, but it wasn't. B reanalyzed the original data and showed that many the obscure naysayers, the ones that the mainstream peers had relegated to obscure journals and un-peer-reviewed working papers were right. The textbooks were wrong. Because of an assumption that was explicit in the derivation of the statistical tools. The PI had no out; all the other mainstream researched mimicked his tools and assumptions. He was wrong and B got his first article published in a mainstream journal as primary author. The PI insisted on it, and B got glowing recommendations. A lesser man would have told B to be quiet; dropped him from the program; or at least tried to hush up something so embarrassing, so overlooked for years and likely to be overlooked for years more. The PI was a good man, not just a good scholar. (People tend to forget this. My first advisor? Let's just echo a colleague who pointed out that you don't have to be a good person to do great research.)
Invariably the problem boils down to assumptions. If they're not explicit and understood, if they're not discussed, there's the risk that inferences and inferred facts will be confused with observed facts. This is a horrible mistake to make and, for the most part, only stupidity results from it. Since people trust themselves they often state something as absolute fact when it is actually inferred. Professors usually suffer from this more often than mere "people."
Take today's NYT. There's a graph showing that 2001-2009 is likely the warmest decade on record. The graph purports to be measured temperatures, or averages of measurements.
The graph lies. Or, more precisely, the graph has inherent in it a number of implicit assumptions so that we layfolk infer something false. If you are used to the assumptions it's not a problem. If you're not, then you take the graph at face value and it appears to lie. It seems to say that it's direct measurements, but the direct measurements are meaningless so that can't be the case.
Why? Because they're made with different equipment, sometimes with different calibration standards, over time. Moreover, since it's a global figure you have to weight each datapoint. Data points change location and you have to interpolate; they're replaced, and you have to say they're somehow equivalent. Then there's the entire proxy business, where you adjust the observations based on the assumed error as revealed by some proxy. It means that what is taken as mere fact is actually the result of some pretty intensive analysis, itself built on analysis, and each analysis has a fair number of assumptions embedded in it. Then we take that output as "observed fact." But it's not a fact. It's a claim, true only (1) if all the data and reasoning behind it is true or (2) it's independently true. (2) is what it's trying to show, (1) isn't shown. It's assumed.
Without discussion of the data and assumptions there can be no valid discussion. We can jump up and down and scream all we want, but we're just saying that we trust the assumptions that others have made. It's easy, and flatters them, but doesn't pass the critical thinking sniff test.
Now, the CRU data dump--most of the dump was data and intermediate analyses--is a problem for precisely this reason. Amateur skeptics and supporters have just focused on the sensationalistic sound-bite emails. It's easy to be dismissive--we have lots of snarky emails, lots of emails actually discussing data and assumptions and analysis, we have chunks of code, gobs of data, and reams of intermediately produced graphs. But let's focus on the snarky emails making up < 1% of the CRU dump.
Making it easier is the fact that some "true skeptics"--and there are such--look at the raw data and ignore the fact that they're unuseable. The NYT won't address one kind of problem because it's too hard; then the gain-sayers simplify quantum physics to, "Well, maybe." Idiocy.
Then there's the second set of true skeptics. They're breaking into two groups, roughly. The first is looking at the data in the dump, and don't like what they find. The discussion of the state of the databases should provide enough soundbites of its own, but it's a "hard topic" to bring up. You can't fit it in 3 paragraphs, of which 1 explains the CRU dump, the second the objection, and the third a plausible excuse. The second group is looking at the interim and partial analyses and trying to sort out what the assumptions are.
To deal with the first group somebody needs to show that the at least partly corrupt datasets weren't used in any published research, or at least none that's important. Given the lack of good will, the way to do that is to make available the datasets that were used. Good luck with that. CRU is in full CYA mode and are less interested in transparency than in advocacy. Never good in science, if you ask me.
Dealing with the second group is rather harder. The CRU folk have essentially said that their datasets are untrustworthy, just piecemeal untrustworthy. I mean, there must be reasons for using tree ring data as proxies here and there but not other places--perhaps the surface measurement data is corrupted? But then why is it suddenly not corrupted at other times? We adjust based on satellite measurements in one spot, but satellite measurements continued past when they were used for adjusting surface measurements--why only use them when they do? If the way that things were measured in 1959 is so horrible, why were they suddenly good to use in 1961--what happened then?
We have error bars, but what do they mean that many layers of adjustment away from the actual raw data? We have an error bar for the average temperature for 1875--where did it come from? Did they actually test the accuracy and precision of measurements on surviving historical instruments? Since most of the world wasn't measured in 1875, how did they extrapolate? Messy, messy--and dripping assumptions, gaps, and problems.
Sometimes it's obvious this is the case. So what was it, 1998 was the warmest year on record for the (continental?) US for years until somebody used the publicly available information to show that there was an error and it was some time in the late 1930s? Then within a fairly short time the NASA crew, I guess it was, found another mistake or recalibrated something to restore 1998 as the warmest year. Notice: The error was ignored until it challenged a claim important for PR; then another error was found to undo the effects of the first error and restore the claim. Did they stop looking after the second error, or simply trust the skeptics to do that work for them? Oddly, I don't even know if that's using raw data, extrapolated data, data adjusted by proxies, or something else. See--it doesn't really matter, the results of extensive analysis are so often taken to be primitives that the question doesn't usually come up. (I'd also point out that the skeptics are no less engaged, all too often, in advocacy science than the supporters are.)
I've heard some answers, but they all have sucked so far. "We've always done it that way." "Well, it's standard." One of the few specific answers, IIRC, was along the lines of "It's statistically significant at the 66th confidence interval." 66th? I hope *that's* not standard.
And then there's the unpleasant "You don't understand"--which is proxy for "we don't think you need to understand, we're here, we're better than you, just do as we say for your own good. What do you think this is, a democracy where we think we need to inform the populace?"
|
Printer-friendly format
Email this thread to a friend
Bookmark this thread
(1000+ posts)
Send PM |
Profile |
Ignore
