New definition for 'peer review'

[guardian]

http://www.dailymail.co.uk/news/article-1254660/Climategate-professor-Phil-Jones-admits-sending-pretty-awful-emails.html

Professor Phil Jones describes the doomer definition of peer review: "it was not 'standard practice' to release data and computer models so other scientists could check and challenge research." Though I believe those outside the Holy Church of AGW still refer to this behavior as 'Groupthink'.

[LeftishBrit]

1. Warning: the Daily Mail is a very unreliable, very RW British rag

See this thread from a couple of years ago. It still applies.

http://www.democraticunderground.com/discuss/duboard.php?az=show_topic&forum=389&topic_id=2174684

[Viking12]

2. The OP has no concern for truth or accuracy

As long as the article fits into his denier world view it can say anything or be from anywhere.

[muriel_volestrangler]

3. No need to warn; the Daily Mail is the thread starter's favourite source

They know exactly what to find in the Daily Mail, and they get what they're looking for.

[guardian]

5. You doomers are ridiculous

Okay a different source: http://www.independent.co.uk/news/science/scientist-admits-leaked-emails-were-pretty-awful-1914295.html

"But it was Professor Jones who held centre stage yesterday and MPs listened intently as he agreed that in the past, climate data, including lists of named weather stations, and the computer code used to analyse the data they provided, had not been made publicly available, in spite of repeated requests. He told the MPs that it was "not standard practice".

Labour MP Graham Stringer said to him: "If that's not standard practice, how can science progress?" Professor Jones replied: "Maybe it should be but it isn't.""


Still doesn't change the point of the OP. These guys wouldn't release data to anyone that wasn't a kool-aid drinking cheerleader for AGW. Yeah that's real rigorous peer review.

[joshcryer]

9. Yes, I am not a fan of Hadley or even the EU way of doing things.

I prefer the way the US does things. If you don't like the closed nature of EU science, then disregard it and use only American science.

GISS releases everyfuckingthing.

[FBaggins]

10. "GISS releases everyf-ingthing."

Now.

[joshcryer]

14. GISTEMP has been available for years.

It wasn't like Gavin magically released it recently to appease people. In fact, the biggest complaint from deniers was that they couldn't compile the code, for years.

[FBaggins]

15. GISTEMP and "everyf-ingthing" are synonyms now?

N/t

[joshcryer]

19. Who said that? NCDC raw was available long before GISTEMP was.

[muriel_volestrangler]

12. It's about your preference for right wing sources

I suspect this is because you are too lazy to read anything other than climate change denier blogs before rushing to DU to repeat whatever they tell you. We're going to have to remind everyone, time and again, that you suck at the teat of right-wing propaganda, by preference.

[wtmusic]

18. The other side of that coin:

They wouldn't release data to morons who weren't capable of understanding it, and would do their best to twist it for ideological purposes.

[joshcryer]

20. Exactly, even with access to all the raw data they "call things in to question."

Often using spurious reasoning and ignorance if not outright lies.

[joshcryer]

4. It wasn't and isn't standard practice to release data and code.

Data usually comes from repositories that all scientists have access too (but not usually the general public; that has changed, though). Code, of course, serves little use, if you're trying to reproduce or expound upon ones given methodology.

There's a reason, say, Steven Macintyre hasn't contributed anything to the scientific process in many years. When he finds "flaws" if those "flaws" were actually scientifically sound, they would be quickly swept up by the scientific community and accepted as flaws. And then corrected.

[FBaggins]

7. That's a bit of an overstatement.

If neither the data nor the methodology is released... what then do "peers" review... the clarity of the writing?

if those "flaws" were actually scientifically sound, they would be quickly swept up by the scientific community and accepted as flaws. And then corrected.

This is likely true, but begs the question as a debate point.

[joshcryer]

8. "methodology" is not "code"

"Methodology" are those algorithms and methods you use, which are always presented in the papers that show results.

"We used such and suchs' analysis method to derive these data."

It is up to other scientists to use "such and suchs'" method to derive the data.

Code sharing does occur, but it is not necessary for the scientific process.

[FBaggins]

11. I think there's more overlap between the two than you credit.

If all you're saying is "we used data you already have and applied methods you've already used" then there isn't anything to review in the first place (i.e., there is no new science). If you rely on the "peers" to produce their own code to implement the defined algorithms, then "peer review" loses much of it's validity, because there's more work than you can expect them to produce just to review an article. They essentially take your word for it.

When you release the code, then a reviewer can simply check whether your code does, in fact, implement the algorithms that you say you're using.

Picking at nits, I know, but this is how it should be done.

[joshcryer]

13. That's not true and it has been done before.

What happens is that a scientist uses their own methods to produce results. If their results do not match the same as others, then they write the authors and get together and discuss it.

See this RealClimate article: http://www.realclimate.org/index.php/archives/2009/12/please-show-us-your-code/

Gavin attempted to reproduce a given result going by the methods presented in a paper, they were unable to because the paper left out important information: http://pubs.giss.nasa.gov/abstracts/2009/Benestad_Schmidt.html

Fortunately, the most accurate temperature record, GISS, does release all of its code and all of its raw data (and I have confirmed that their data comes directly from the NCDC). So we don't have any real squabble. It's simply that if a method is released and all algorithms are covered, you should be in the clear, especially if you want to, as the EU guys like to do, keep your code proprietary for future research (say you weren't done using your code to make papers, for instance, and didn't want others replicating your code in order to beat you to the punch).

I do hope transparency comes about in due time, within the scientific community, but I'm not sweating it.

[FBaggins]

17. I disagree

What happens is that a scientist uses their own methods to produce results.

Peer review has to allow others to reproduce your results. There has to be enough "there" there so that they aren't starting from scratch (because if they are, that's their own publishable work).

Fortunately, the most accurate temperature record, GISS, does release all of its code and all of its raw data (and I have confirmed that their data comes directly from the NCDC). So we don't have any real squabble.

Oh I'm not squabbling about the scientific consensus... merely the conduct of a few of those scientists.

[joshcryer]

23. If there isn't "enough there" in the methods presented in a paper...

...then that paper should not pass the peer review. If scientists relied on the work of others, the actual code, then they could overlook a bug in that code. This is why, instead of there being one climate model, there are dozens. If you had just one climate model then that model could be faulty and the whole of the scientific literature would be compromised.

[Statistical]

16. I disagree. Open source code allows you to verify there are not flaws in the calculation.

As I have posted in other threads everything should be open. From raw data (if it is analog, paper, records, etc) then it should be scanned, the compiled data, the source code, the compiled programs, the algorithms, the proofs to support those algorithms, the assumptions, the output, the projections.

EVERY SINGLE THING SHOULD BE PUBLICLY AVAILABLE.

Open source has been a huge success in the software world because no matter how smart you are you can make mistakes, either careless mistakes that are missed, or mistakes in methodology.

The sooner the IPCC and other Climate Change organizations realize this the more effective they will be.

While if I have the data and algorithm I could write my own code to do the calculations but what happens if the results don't match.
Is it because their results are falsified?
Is it because their results are based on flawed code?
Is it because I misunderstood the methodology/algorithm?
Is it because I made a mistake in my code?

Having the input and the output and being told "trust us the code is good" is not going to improve public support. It is possible the methodology is sound and the code if flawed.

[joshcryer]

21. I agree, but it wasn't necessary for the peer review in the past.

And it's not in the future.

It's a nice plus though.

[joshcryer]

22. For an example, GISS ModelE is open source. Except for some small parts.

Those small parts are the aspect of several scientific publications that are to be released. They will not release the source code for those parts until those papers are complete, and peer reviewed.

Peers do not have to agree with what you say, they need only accept that what you say has a sound basis and is represented by your arguments, and by your methods.

You can recreate GISTEMP simply by going by Hansen et al (2001). It would be difficult, of course, since it's thousands of lines of FORTRAN code.

While if I have the data and algorithm I could write my own code to do the calculations but what happens if the results don't match.
Is it because their results are falsified?
Is it because their results are based on flawed code?
Is it because I misunderstood the methodology/algorithm?
Is it because I made a mistake in my code?

Science, it's a bitch. It could be any of those things. You can ask the scientist in charge of the code to let you look at it. At least in the case of GISS all of the code is made available. But the peer review is not a perfect case of absolute verification, it is to determine if arguments are sound. Further analysis by the body of scientific research finds holes in ones observations, not a panel of guys looking over your paper to make sure it has a sound basis.

[Statistical]

24. That is my point.

Having all information (raw data, methodology, source code, theory, assumptions, etc) available for published results makes the system more accurate.
I have no problem with people not providing data for unpublished results.
I just think in general the IPCC needs to adopt an more "open-source" model for scientific review and scrutiny.

I don't claim to have all the answers but the sliding public confidence in global warming indicates the IPCC hasn't handled this very well.

[joshcryer]

26. I agree and I think the climate scientists are coming around to that.

Certainly the American's have been providing it for 3 years now, long before the denialosphere got so big.

[joshcryer]

27. But, as far as losing the PR war, blame the snow, blame liars, don't blame the scientists.

For example, someone here posted a graph taken from this page: http://joannenova.com.au/2010/02/the-big-picture-65-million-years-of-temperature-swings

It is outright lying. I looked at the ice core graph and the data it was derived from. The data ends 122 years ago. They're lying by saying that the data puts it in to "historical context."

Then you look at the sediment data, and again, it cannot be representative of the last 2 thousand years, and indeed, has such a large sampling period that it is effectively irrelevant for significant temperature trends. But they leave that out.

It's just egregious, it's downright lying.

But there's really nothing scientists can do about it. It's moon hoax level stuff. It's creationism level stuff. It takes quite awhile to become irrelevant.

[joshcryer]

28. Here's a great post why replication, not repetition, is good:

http://tamino.wordpress.com/2010/03/01/replication-not-repetition/

[Nihil]

29. That is the simplest response to the "groupthink" attacks ...

> Peers do not have to agree with what you say, they need only accept
> that what you say has a sound basis and is represented by your arguments,
> and by your methods.

The importance of "peer review" is that it ensures consistency of your case,
not that everyone agrees with what you are saying.

Thanks for that!
:hi:

[kristopher]

25. Do you really think so?

Where, precisely, do you propose that code be published? Do you expect the journals to provide the pages?

Things like this are why any journal article has a lead author with contact information provided. If someone has a question they contact the author and the author might have the code or they might not. Things often get thrown away, lost or mislaid after a couple of years. In the article their method will be detailed to allow others to reproduce their work by writing their own code if the original is unavailable. If the original is available it is usually shared on request.

Publishing that kind of ingredient to a research effort may be a bit more practical with electronic publishing, and in fact if you look you'll see many "supplemental data" files appended to electronic articles. But that has nothing to do with the origins of this contrived controversy and it would be absurd to expand that to include every piece of code that was part of the research. Do you really think people are going to be combing over all the code in the thousands of articles published monthly? Really do you think that?

I think the present system is doing a pretty good job and the only people who are really worried about it are those that either don't understand it or are responding in a timid manner to those that don't understand it.

Your two concepts are both already in place - "open source" creativity is already part of peer review. When an article is written it is very, very seldom the product of one person. Even if others aren't listed as authors there is considerable involvement by other researchers in polishing and checking the work in progress. That part is very similar to what you are relating to open source for software; it is also standard for academia where you have ready access to experts of all disciplines. Where, after all, do you think the concept came from in the software industry?

So that is *part* of the peer review process, with more coming at later stages where the whole of the research is tested against the body of knowledge as seen by authorities in the topic of the paper. And if there is a problem, then they will look to things like the code to see if it lies there. If no questions arise, things created just for that effort are bound to go the way of other clutter; because after all, if needed the original paper has enough information to allow the same effort to be repeated.

[On the Road]

6. There May Indeed Have Been Groupthink

but the article doesn't clarify what things about the data and modeling were not publicized at what point.

Peer review takes places when articles are published. The majority of scientific work goes on outside of the publication process. There is nothing wrong with spending time developing a model and collecting data and sharing it only when it is published and enters the public sphere.

Some of the criticism of Jones's modeling centered around the fact that data from northeast China did not indicate whether the location of tracking stations had changed over time. That is not consistent with charges of a secret, biased study process.