Last week, I wrote about a scientific paper that was published in the elite journal Nature in 1995. Within a couple of years, the findings of said paper were called into question by several other papers in different journals. As of today, nearly two decades since the original came out, nobody has replicated it. And yet, it’s still sitting there in the literature, still influencing others. It’s been cited nearly 1,000 times.
Some readers were angry with my post, arguing, for example, that “science’s self-correcting paradigm works over decades”. Indeed, that was my point. Science’s self-correction is generally very slow — perhaps, as many argue, too slow.
This week I learned about an unfolding scientific debate that’s got me thinking again about the challenge — the impossibility? — of swift and sure scientific correction. What does it mean when one group of researchers, or even two or three groups, can’t replicate a particular scientific finding? Does that necessarily mean it’s wrong? At what point should a scientist give up on a new idea for lack of supporting evidence?
That unfolding debate started in late 2011, when Chen-Yu Zhang’s team from Nanjing University in China found something pretty wild: bits of rice RNA floating in the bloodstreams of Chinese men and women. That might not seem so strange; rice was a primary ingredient of their diets, after all. But RNA molecules are pretty fragile. So the discovery shocked and intrigued many biologists.
“It’s just a very neat new physiologic mechanism,” says Ken Witwer, a molecular biologist at Johns Hopkins University in Baltimore. “How is it that a small RNA, or any RNA, could survive this trip from the mouth, with all these enzymes in saliva, down into the stomach, with the acidic environment there, and make it all the way into the gut, to the point that it could cross over into the blood? What form would this RNA have to be in to make that journey?”
Even more provocative: Zhang’s study also showed that in mice, those same tiny pieces of plant RNA — dubbed microRNA or miRNA, and made up of just two-dozen nucleotides, or letters of code — can shut down a gene involved in cholesterol uptake.
The study had big implications for medicine and our food supply. For instance, it suggested that researchers might be able to design oral RNA drugs for a host of diseases, “one of the holy grails” of the field, Witwer says. The data also provided evidence, at least according to a press release issued by Zhang, that miRNAs are “essential functional molecules” in Chinese herbal remedies. Finally, some people — like the author of a controversial* column published in The Atlantic — used the study to argue that genetically modified organisms (GMOs) are harmful to eat (despite loads of evidence to the contrary). (Update 7/9: See below a response from the author of that column.)
A few of those could be considered replication studies. In one, David Galas of the Pacific Northwest Diabetes Research Institute, in Seattle, performed genetic sequencing of human blood samples and found low levels of miRNA from many species, including bacteria, fungi, insects, and plants. Galas’s team detected the same specific rice miRNA that Zhang had — dubbed miR-168 — albeit at far lower levels than Zhang had.
Two other follow-up studies were bankrolled by agricultural giant Monsanto (which, it must be said, sells GMOs and thus has a big stake in claims that they’re dangerous). The Monsanto researchers combed through large datasets of genetic sequences obtained from mammals, chickens, and insects, looking for any trace of plant miRNAs. They found them in some of the datasets, but again, at very low levels. And sometimes the data didn’t make sense — they found miR-168, for example, in animals that had never eaten food containing miR-168, suggesting that it could have been the result of a contamination, Witwer says. “We know that pollen has miRNAs in it, and depending on the time of the year, maybe we have more pollen contamination, even in our best labs, than at other times.”
The July issue of RNA Biology adds two more skeptical papers to the mix. In one of them, Witwer’s team fed monkeys a Silk fruit and protein shake, which happens to contain high levels of miR-168 and other plant miRNAs. The researchers tested the animals’ blood for miRNAs before the feeding and 1, 4, and 12 hours after the feeding.
The scientists used the same method that Zhang’s group had: polymerase chain reaction, or PCR, which allows researchers to identify specific segments of DNA or RNA by copying them over and over again, and then fluorescing the copies. When Witwer’s team used PCR to find miRNAs in the smoothies, the results were sensitive and consistent. But when looking at the monkeys’ blood, the PCR data were much more variable. “We weren’t completely confident in the accuracy of the method,” Witwer says.
So his team repeated the experiment using a newer and more precise type of PCR, called droplet digital PCR. This time, they again saw a lot of variability in the blood data, and no consistent differences between the samples taken before and after the animals ate the shakes. Witwer’s conclusion: Plant miRNAs probably don’t transfer into our blood after digesting it, at least not in quantities anywhere near what Zhang’s group had reported.
In the other new paper, Stephen Chan of the Brigham and Women’s Hospital in Boston found that healthy athletes did not carry detectable levels of plant miRNAs in their blood after eating fruit chock-full of those molecules. The scientists also couldn’t find this kind of transfer in experiments with mice and bees. “We conclude,” the paper states, “that horizontal delivery of microRNAs via typical dietary ingestion is neither a robust nor a frequent mechanism.”
So what do all of these studies say about this particular finding, and more generally, about science’s self-correcting process?
Less than two years after the original paper came out, at least five studies have followed it up. And in my (utterly non-expert) judgment, it seems like none of them meaningfully replicate Zhang’s paper. (
Zhang has not responded to my request for comment; I will update the post if/when he does. Update, 7/8: Zhang has responded to my request for comment; see his full response at the bottom of this post.)
The studies are consistent in finding very low levels of plant miRNAs in people and a variety of other species. Witwer says that’s enough evidence of a non-result to move on from the whole idea. ”I’m willing to help out if someone’s organizing an attempt to replicate something, but I’m probably not going to devote my lab to answering more questions on this issue,” he says. “We’ve convinced ourslves that we’re not seeing anything here.”
Others, though, aren’t ready to drop it. Galas, whose paper found miR-168 in low levels in human blood, says the only thing we know for sure is how difficult the question is to study. “The major result is that miRs are difficult to measure accurately,” he says. What’s more, he says, Witwer’s feeding experiments aren’t necessarily damning because their specifics differ from the original Zhang paper.
For Galas, the current data only makes the question more worthy of study by the RNA community, not less. “This is a an important topic to get pinned down — the potential for new biological phenomena is significant.”
This story helps explain why science’s self-correction process can’t be super-quick. It takes time for evidence to accumulate and show clear trends. That said, scientists could be better at making that correction process more efficient. One step, Witwer says, is transferring published data into public repositories that can be easily shared with the scientific community.
As Witwer reported in February, less than 40 percent of studies reporting microRNA sequencing data submitted that data to public databases. More interesting: The scientists who did share were more likely to have high-quality papers. The only paper in the analysis to be retracted, by the way, was one that did not share its raw data.
“I think that science can be self-correcting,” Witwer says, “but it requires people to do that correcting.”
*That column was rightfully struck down by science bloggers Emily Willingham and Christie Wilcox, and because of their posts, the author eventually amended it. The self-correction of the blogosphere is just a tad faster than the self-correction of science, eh? (UPDATE 7/9: The author of the column says his re-write had nothing to do with the bloggers; see his full comment here.)
UPDATE #1 (7/4): Also, just noticed that the incomparable Willingham beat me to this story a couple of weeks ago! Go check out her post at Forbes.
UPDATE #2 (7/8): Dr. Zhang sent me a lengthy letter in response to my request for comment about Dr. Witwer’s new study. You can read that (in .pdf form) by clicking here.