Science

SorellaLaBefana

(337 posts) Sat Apr 26, 2025, 03:59 PM Saturday

Another, well-done, failure of replication study discussed in Nature--this one focused on basic methods

Here's what Nature had to say (paywalled)
Huge reproducibility project fails to validate dozens of biomedical studies

...The teams were able to replicate the results of less than half of the tested experiments. That rate is in keeping with that found by other large-scale attempts to reproduce scientific findings. But the latest work is unique in focusing on papers that use specific methods and in examining the research output of a specific country, according to the research teams...

They ended up selecting three of these methods: an assay of cell metabolism, a technique for amplifying genetic material and a type of maze test for rodents. Then the authors randomly selected biomedical papers that relied on those methods and were published from 1998 to 2017 by research teams in which at least half the contributors had a Brazilian affiliation....

The authors judged a paper’s replicability by five criteria, including whether at least half of the replication attempts had statistically significant results in the same direction as the original paper. Only 21% of the experiments were replicable using at least half of the applicable criteria...

Marcelo Mori, a biologist at the State University of Campinas in Brazil, agrees. “Exact replication of an experiment is particularly challenging in the life sciences,” he says. “Living organisms, such as rodents and cell cultures, can respond differently when exposed to environmental variations like temperature, diet, microbiota, or culture medium composition.” Reagents, such as antibodies, can exhibit batch-to-batch variation, he adds...

https://www.nature.com/articles/d41586-025-01266-x?utm_source=Live+Audience&utm_campaign=88bbdfbc5f-nature-briefing-daily-20250425&utm_medium=email&utm_term=0_b27a691814-88bbdfbc5f-49942924

Very much doubt that it reflects anything unique about the country where the study was done—which the Nature article takes pains to explicitly state. Further it mentions "Publish or Perish" as a problem in all countries. Original study (still a preprint) is Amaral, O. B. et al. Preprint at bioRxiv (2025)
https://doi.org/10.1101/2025.04.02.645026

Yet another reminder that, even with an apparently well done study, in clinical medicine it is, in my long held opinion, foolish and potentially unsafe to make practice changes based on only ONE study. Nonetheless, it has been my observation that this is quite often what happens.

3 replies

= new reply since forum marked as read

Highlight:

Another, well-done, failure of replication study discussed in Nature--this one focused on basic methods (Original Post) SorellaLaBefana Saturday OP

Science is two steps forward, 1.9 steps backwards. Always a work in progress. Thanks for bringing this. Sobering. . . nt Bernardo de La Paz Saturday #1

mouse models especially IbogaProject Saturday #2

i was bitten by a single study recently. mopinko Saturday #3

Bernardo de La Paz

(54,988 posts)

1. Science is two steps forward, 1.9 steps backwards. Always a work in progress. Thanks for bringing this. Sobering. . . nt

Reply to SorellaLaBefana (Original post)

Sat Apr 26, 2025, 04:26 PM

Saturday

IbogaProject

(4,306 posts)

2. mouse models especially

Reply to Bernardo de La Paz (Reply #1)

Sat Apr 26, 2025, 05:02 PM

Saturday

Did they do a reddo of the "give rats unlimited access to drugs" and if the environment wasn't bare and had stuff to do and other mice to interact with they don't even use drugs, let alone abuse. And those diets are another area that really alters outcomes, as many of those special diets are missing key nutrients, which help simulate worst case effects but can lead to things like the saccharine scare. And the publish or perish system actually rewards fraud as it is easier to hit the mark often, where legitimate honest research will fail quite a bit.

mopinko

(72,345 posts)

3. i was bitten by a single study recently.

Reply to SorellaLaBefana (Original post)

Sat Apr 26, 2025, 08:22 PM

Saturday

so, there was a study in japan on ambien that found a high risk of falls and fractures in older ppl.
i have taken it b4, it worked well for me. i have an occasional bit of insomnia so i asked for a rx, and got a no. this study was posted here. it was a small study of over 80yo fragile ppl in a nursing home. but somehow here it became anyone over 70.

i was in the hospital recently and had some trouble getting a grip on the pain. so i ended up on a mix of 4 drugs. something in that mix had me revved up and unable to sleep, and going a bit nuts. i asked for and ambien and was told no, cuz there were ‘risks’. what risks? the attending wouldnt even come or talk to me at all. i made them call the pharmacy, and no 1 knew. i kinda knew that was what it was. i wasnt getting out of bed by myself at that point. and i am NOT fragile. my bones r pretty good. there was 0 reason to tell me no.
but all they had to hear was ‘risk’. i got a melatonin! bfd.
imma die mad about that.

Reply to this discussion