or Why Pilot Studies Are Mostly Meaningless (Unless They're Used To Promote "Alternative/Complementary/Integrative/Quack" Treatments.
http://www.sciencebasedmedicine.org/index.php/the-value-of-replication/"...
This episode illustrates potential problems with the scientific literature. We often advocate at SBM that individual studies can never be that reliable — rather, we need to look at the pattern of research in the entire literature. That means, however, understanding how the scientific literature operates and how that may create spurious artifactual patterns.
For example, I recently wrote about the so-called “decline effect” — a tendency for effect sizes to shrink or “decline” as research on a phenomenon progresses. In fact, this was first observed in the psi research, as the effect is very dramatic there — so far, all psi effects have declined to non-existence. The decline effect is likely a result of artifacts in the literature. Journals are more inclined to publish dramatic positive studies (“theory-advancing research”), and are less interested in boring replications, or in initially negative research. A journal is unlikely to put out a press release that says, “We had this idea, and it turned out to be wrong, so never-mind.” Also, as research techniques and questions are honed, research results are likely to become closer to actual effect sizes, which means the effect of researcher bias will be diminished.
If the literature itself is biased toward positive studies, and dramatic studies, then this would further tend to exaggerate apparent phenomena — whether it is the effectiveness of a new drug or the existence of anomalous cognition. If journals are reluctant to publish replications, that might “hide the decline” (to borrow an inflammatory phrase) — meaning that perhaps there is even more of a decline effect if we consider unpublished negative replications. In medicine this would be critical to know — are we basing some treatments on a spurious signal in the noise of research.
....
All of this is happening at a time when prior probability (scientific plausibility) is being given less attention than it should, in that highly implausible notions are being seriously entertained in the peer-reviewed literature. Bem’s psi research is an excellent example, but we deal with many other examples frequently at SBM, such as homeopathy and acupuncture. Current statistical methods and publication biases are not equipped to deal with the results of research into highly implausible claims. The result is an excess of false-positive studies in the literature — a residue that is then used to justify still more research into highly implausible ideas. These ideas can never quite reach the critical mass of evidence to be generally accepted as real, but they do generate enough noise to confuse the public and regulators, and to create an endless treadmill of still more research.
...."--------------------------------
This is definitely a piece that needs to be read in total. It also discusses information that is clearly missing from many discussions on the health forum, and at DU (which I love!), in general.
Printer-friendly format
Email this thread to a friend
Bookmark this thread
(1000+ posts)
Send PM |
Profile |
Ignore
