Another, well-done, failure of replication study discussed in Nature--this one focused on basic methods [View all]
Here's what Nature had to say (paywalled)
Huge reproducibility project fails to validate dozens of biomedical studies
...The teams were
able to replicate the results of less than half of the tested experiments. That rate is in keeping with that found by other large-scale attempts to reproduce scientific findings. But the latest work is unique in focusing on papers that use specific methods and in examining the research output of a specific country, according to the research teams...
They ended up s
electing three of these methods: an assay of cell metabolism, a technique for amplifying genetic material and a type of maze test for rodents. Then the authors randomly selected biomedical papers that relied on those methods and were published from 1998 to 2017 by research teams in which at least half the contributors had a Brazilian affiliation....
The
authors judged a papers replicability by five criteria, including whether at least half of the replication attempts had statistically significant results in the same direction as the original paper. Only 21% of the experiments were replicable using at least half of the applicable criteria...
Marcelo Mori, a biologist at the State University of Campinas in Brazil, agrees. Exact replication of an experiment is particularly challenging in the life sciences, he says. Living organisms, such as rodents and cell cultures, can respond differently when exposed to environmental variations like temperature, diet, microbiota, or culture medium composition. Reagents, such as antibodies, can exhibit batch-to-batch variation, he adds...
https://www.nature.com/articles/d41586-025-01266-x?utm_source=Live+Audience&utm_campaign=88bbdfbc5f-nature-briefing-daily-20250425&utm_medium=email&utm_term=0_b27a691814-88bbdfbc5f-49942924
Very much doubt that it reflects anything unique about the country where the study was donewhich the Nature article takes pains to explicitly state. Further it mentions "Publish or Perish" as a problem in all countries. Original study (still a preprint) is Amaral, O. B. et al. Preprint at bioRxiv (2025)
https://doi.org/10.1101/2025.04.02.645026
Yet another reminder that, even with an apparently well done study, in clinical medicine it is, in my long held opinion, foolish and potentially unsafe to make practice changes based on only ONE study. Nonetheless, it has been my observation that this is quite often what happens.