A Blog by Virginia Hughes

So Science…Might Have Gotten It Wrong. Now What?

Last week, I wrote about a scientific paper that was published in the elite journal Nature in 1995. Within a couple of years, the findings of said paper were called into question by several other papers in different journals. As of today, nearly two decades since the original came out, nobody has replicated it. And yet, it’s still sitting there in the literature, still influencing others. It’s been cited nearly 1,000 times.

Some readers were angry with my post, arguing, for example, that “science’s self-correcting paradigm works over decades”. Indeed, that was my point. Science’s self-correction is generally very slow — perhaps, as many argue, too slow.

This week I learned about an unfolding scientific debate that’s got me thinking again about the challenge — the impossibility? — of swift and sure scientific correction. What does it mean when one group of researchers, or even two or three groups, can’t replicate a particular scientific finding? Does that necessarily mean it’s wrong? At what point should a scientist give up on a new idea for lack of supporting evidence?

That unfolding debate started in late 2011, when Chen-Yu Zhang’s team from Nanjing University in China found something pretty wild: bits of rice RNA floating in the bloodstreams of Chinese men and women. That might not seem so strange; rice was a primary ingredient of their diets, after all. But RNA molecules are pretty fragile. So the discovery shocked and intrigued many biologists.

“It’s just a very neat new physiologic mechanism,” says Ken Witwer, a molecular biologist at Johns Hopkins University in Baltimore. “How is it that a small RNA, or any RNA, could survive this trip from the mouth, with all these enzymes in saliva, down into the stomach, with the acidic environment there, and make it all the way into the gut, to the point that it could cross over into the blood? What form would this RNA have to be in to make that journey?”

Even more provocative: Zhang’s study also showed that in mice, those same tiny pieces of plant RNA — dubbed microRNA or miRNA, and made up of just two-dozen nucleotides, or letters of code — can shut down a gene involved in cholesterol uptake.

The study had big implications for medicine and our food supply. For instance, it suggested that researchers might be able to design oral RNA drugs for a host of diseases, “one of the holy grails” of the field, Witwer says. The data also provided evidence, at least according to a press release issued by Zhang, that miRNAs are “essential functional molecules” in Chinese herbal remedies. Finally, some people — like the author of a controversial* column published in The Atlantic — used the study to argue that genetically modified organisms (GMOs) are harmful to eat (despite loads of evidence to the contrary). (Update 7/9:  See below a response from the author of that column.)

Andres Rodriguez, via Flickr
Andres Rodriguez, via Flickr

So the paper made its media splash. And in the 21 months since its publication, the work has been cited in 42 other papers, according to Web of Knowledge.

A few of those could be considered replication studies. In one, David Galas of the Pacific Northwest Diabetes Research Institute, in Seattle, performed genetic sequencing of human blood samples and found low levels of miRNA from many species, including bacteria, fungi, insects, and plants. Galas’s team detected the same specific rice miRNA that Zhang had — dubbed miR-168 — albeit at far lower levels than Zhang had.

Two other follow-up studies were bankrolled by agricultural giant Monsanto (which, it must be said, sells GMOs and thus has a big stake in claims that they’re dangerous). The Monsanto researchers combed through large datasets of genetic sequences obtained from mammals, chickens, and insects, looking for any trace of plant miRNAs. They found them in some of the datasets, but again, at very low levels. And sometimes the data didn’t make sense — they found miR-168, for example, in animals that had never eaten food containing miR-168, suggesting that it could have been the result of a contamination, Witwer says. “We know that pollen has miRNAs in it, and depending on the time of the year, maybe we have more pollen contamination, even in our best labs, than at other times.”

The July issue of RNA Biology adds two more skeptical papers to the mix. In one of them, Witwer’s team fed monkeys a Silk fruit and protein shake, which happens to contain high levels of miR-168 and other plant miRNAs. The researchers tested the animals’ blood for miRNAs before the feeding and 1, 4, and 12 hours after the feeding.

The scientists used the same method that Zhang’s group had: polymerase chain reaction, or PCR, which allows researchers to identify specific segments of DNA or RNA by copying them over and over again, and then fluorescing the copies. When Witwer’s team used PCR to find miRNAs in the smoothies, the results were sensitive and consistent. But when looking at the monkeys’ blood, the PCR data were much more variable. “We weren’t completely confident in the accuracy of the method,” Witwer says.

So his team repeated the experiment using a newer and more precise type of PCR, called droplet digital PCR. This time, they again saw a lot of variability in the blood data, and no consistent differences between the samples taken before and after the animals ate the shakes. Witwer’s conclusion: Plant miRNAs probably don’t transfer into our blood after digesting it, at least not in quantities anywhere near what Zhang’s group had reported.

In the other new paper, Stephen Chan of the Brigham and Women’s Hospital in Boston found that healthy athletes did not carry detectable levels of plant miRNAs in their blood after eating fruit chock-full of those molecules. The scientists also couldn’t find this kind of transfer in experiments with mice and bees. “We conclude,” the paper states, “that horizontal delivery of microRNAs via typical dietary ingestion is neither a robust nor a frequent mechanism.”

Forest Wander, via Flickr
Forest Wander, via Flickr

So what do all of these studies say about this particular finding, and more generally, about science’s self-correcting process?

Less than two years after the original paper came out, at least five studies have followed it up. And in my (utterly non-expert) judgment, it seems like none of them meaningfully replicate Zhang’s paper. (Zhang has not responded to my request for comment; I will update the post if/when he does. Update, 7/8: Zhang has responded to my request for comment; see his full response at the bottom of this post.)

The studies are consistent in finding very low levels of plant miRNAs in people and a variety of other species. Witwer says that’s enough evidence of a non-result to move on from the whole idea. “I’m willing to help out if someone’s organizing an attempt to replicate something, but I’m probably not going to devote my lab to answering more questions on this issue,” he says. “We’ve convinced ourslves that we’re not seeing anything here.”

Others, though, aren’t ready to drop it. Galas, whose paper found miR-168 in low levels in human blood, says the only thing we know for sure is how difficult the question is to study. “The major result is that miRs are difficult to measure accurately,” he says. What’s more, he says, Witwer’s feeding experiments aren’t necessarily damning because their specifics differ from the original Zhang paper.

For Galas, the current data only makes the question more worthy of study by the RNA community, not less. “This is a an important topic to get pinned down — the potential for new biological phenomena is significant.”

This story helps explain why science’s self-correction process can’t be super-quick. It takes time for evidence to accumulate and show clear trends. That said, scientists could be better at making that correction process more efficient. One step, Witwer says, is transferring published data into public repositories that can be easily shared with the scientific community.

As Witwer reported in February, less than 40 percent of studies reporting microRNA sequencing data submitted that data to public databases. More interesting: The scientists who did share were more likely to have high-quality papers. The only paper in the analysis to be retracted, by the way, was one that did not share its raw data.

“I think that science can be self-correcting,” Witwer says, “but it requires people to do that correcting.”

*That column was rightfully struck down by science bloggers Emily Willingham and Christie Wilcox, and because of their posts, the author eventually amended it. The self-correction of the blogosphere is just a tad faster than the self-correction of science, eh? (UPDATE 7/9: The author of the column says his re-write had nothing to do with the bloggers; see his full comment here.)

UPDATE #1 (7/4): Also, just noticed that the incomparable Willingham beat me to this story a couple of weeks ago! Go check out her post at Forbes.

UPDATE #2 (7/8): Dr. Zhang sent me a lengthy letter in response to my request for comment about Dr. Witwer’s new study. You can read that (in .pdf form) by clicking here.

24 thoughts on “So Science…Might Have Gotten It Wrong. Now What?

  1. The idea that science is self-correcting is like the idea that economic systems are self-correcting, or the idea that democratic political systems are self-correcting, or the idea that corporate behavior is self-correcting, or that religious institutions are self-correcting. They are until they aren’t. Science is probably better at self-correction than the other examples, but of course there will be egregious failures from time to time. It is like the teleological notion that evolution always results in organisms with higher fitness. Sometimes it does, but sometimes it doesn’t. There is no mystical force guiding any of these things.

  2. One more thought. Say you get some spinach that is contaminated witha small amount of pathogenic E. coli, and feed it to a bunch of persons, then test their blood for the presence of E. coli. Some of those persons will have no E. Coli in their blood, some will have a little bit, some may have a lot. Some of the persons with a little, will get sick; others will not. You observe two facts: the amount detected is variable (usually quite small), and that the outcome is not strictly determined by the amount detected. Those facts fail to prove the non-pathogenicity of the E. Coli. All it shows is that people are variable, and that complex systems are hard to study.

  3. Very nice. The other big half of the self-correction story is self-expansion. Good science grows.

    When a paradigm shifting discovery is made the lab that makes it usually has the next study, the next paper, the next grant well underway before the original work his press. You have a lead and you must move fast! The best science has a strong corollary that expands on the original finding.

    After that, science grows. In the days of high-throughput sequencing and proteomics providing massive datasets in weeks, hundreds of other labs probably sequenced blood serum, or hammered public datasets (which are huge) looking for evidence of an miRNA corresponding to their favorite gene.

    I was so intrigued when this paper came out and really wanted it to be true, still do. It would open a new area of inquiry, which is always cool. I told my class, “Remember this one and let’s see if the next work takes this finding to a new level.”

    We’re still waiting.

  4. The notion that science (or economics) inevitably corrects itself is a rhetorical tautology and a myth that cannot be disproved, Popperian style: it’s basically a nonsense.

  5. Person does a study. Publishes it. Others can’t replicate it. They publish that. Looks like the system’s working. What’s your point? By the way, where’s the bit in your article where you address the “now what” referred to in the title?

  6. Yes, science is self-correcting. The correction sometimes takes years to take effect. But these error happen (assuming no intention to defraud) because before before sending the paper to the editor and referees, the investigator didn’t make sure that did robust experiments that could falsify his own results. The scientists as to be his own worst critic.

  7. A couple of commenters here claim science (and some other things) aren’t self-correcting. In fact, everything which is subject to selection is constantly corrected according to the premise of the selection. Evolution creates higher fitness because the fit survive more often. If the scientific environment sometimes doublechecks studies and discards the results that can’t be replicated, then the results that fit reality better survives and the scientific body of knowledge evolves. This process is not a myth, nor does it require some mystical force.

  8. Angela Kennedy, I could be confused about your claim about Popper or perhaps you got the Philosopher confused (Kuhn, Feyerabend?). At least from my interpretations, Popper argued successfully that falsifiability is what allows science to be successful.
    In fact, from copying from the hard sciences control and double-blind experiments, the applied and social sciences have produced more reliable data.
    However, scientists are human and their shortcomings and biases can ruin the check and balances of the scientific method. The mistake that is hard to avoid is blaming the flaws on the scientific method, instead of human error.
    Joseph also makes a more broad claim that includes democracy. Democracies can fail, but while more dependent on probability and manipulation, they do self-correct over a longer period of time. Not good enough? Give me an improvement to the one we have.

  9. Science is a process. The process has proceeded. As it should be.

    What will be interesting is to see if any of the folks who ran with the fearmongering initially–Ari LeVaux, David Icke, Alex Jones, Mike Adams, Jack Heinemann, etc–will choose to write about the follow-on work.

    And that’s the difference between science and anti-science.

  10. Mary, the question is: will the researchers be funded to do any follow on? Happy to see science proceed, and self correct, but how about keeping the food science in the lab until the process is finalised.

  11. Why publish before all the checking and correcting is done? A big reason: we never know when all the needed checking is done. And no one lab can do all the checking — it may be making a systematic error. Getting the information out there is necessary to start the error-correcting process. Not to mention to promote one’s career and lay groundwork for getting the funding necessary to do follow-up work.

  12. Virginia,

    This is Ari LeVaux. I wrote the story that was posted to The Atlantic Online, linked above, which you characterized as using the Zhang study “…to argue that genetically modified organisms (GMOs) are harmful to eat.”

    Given your interest in self-correction, I wanted to share here the comments that I posted to Emily Willingham’s recent piece about that story. Some of these apply to your story as well, and I look forward to seeing how you update it.

    Thanks for considering my comments,



    My last name is spelled LeVaux, not Levaux.

    As my “scorned” column clearly stated, in both the original and the rewrite, if Zhang’s results survive scientific scrutiny then they could have huge implications. Since then I’ve reminded you by email that I had made this key point, which in the re-write I had stated as:

    “Should the research survive scientific scrutiny — a serious hurdle — it could prove a game changer in many fields.”

    Given this qualification, I don’t see how you can characterize my argument as “leaping.”

    Also, you should know that despite what you may have read in the Atlantic, you and Christie Wilcox had nothing to do with inspiring my re-write of this story. In fact, if you look at the dates on Wilcox’s rebuttal (January 12) and my re-write on Alternet (January 11), you’ll see that, barring time travel, it was impossible for her to have had anything to do with it. As for your role, Alternet had already agreed to post my re-write before I knew who you were, or had read the first of your many rebuttals.

    The Atlantic had posted my Alternet re-write without my or Alternet’s permission. After the Atlantic editor who did this apologized to Alternet, he was given permission to leave it up. He also erroneously credited you and Wilcox for forcing me into doing the re-write. I’m not sure why The Atlantic has not changed this confusing and false statement, and given that it remains up there I understand your confusion. But if you are truly concerned with setting the record straight you might as well let that myth die.

    Also, as you know, I had nothing to do with the title that the Atlantic ran: “The Very Real Danger of Genetically Modified Organisms.” I’ve already told you that, and you made note of it in the first of your rebuttals, yet you keep rubbing my face in it anyway, perhaps out of spite or because you feel it strengthens your case. You and others have taken this headline as the essence of my argument, claiming that I argued GMOs are inherently dangerous. What I argued is that it’s conceivable that some constructs could be dangerous, and I questioned Monsanto’s claim that safety testing isn’t necessary. For what it’s worth, I lobbied hard to change that headline from the moment I saw it, to no avail. Since then, the headline has brought me more grief than the article itself.

    Rather than trying to prove that GMOs are dangerous, my article was about the regulatory standard of substantial equivalence, and I used the Zhang paper as an example for why I think this standard is flawed. But the validity of Zhang’s conclusions are not a load-bearing part of that argument. If the Zhang paper does not survive scientific scrutiny, I don’t see what there is for me to address or apologize for, as some in the comments section here and elsewhere have suggested.

    Looking back, I definitely could have made that column a lot less shrill, and wish I had. But the mean-spirited arrogance and stubborn factual errors that have haunted me since I wrote that column have been depressing and ironic. From you, this assault has felt particularly personal. You’ve written at least three articles about my piece, all of which have clung to your original misread. I think it’s time for you to let it go. But before you do, it would be refreshing to see you make an attempt to address the inaccuracies you’ve perpetuated. I won’t hold my breath.

    Ari LeVaux

  13. Apologies for spelling your name without the correct capitalization.

    The first post I wrote was directly in response to your original article. The second one was invited by an editor at Slate. The third is the Forbes article, which is a follow-up to both the Zhang paper and your commentary. There is nothing personal here, and it is not an “assault.” If I recall correctly, we did email during that time, and part of the substance of that discussion related to your rewrite in progress. Regarding the rest, any interested readers can access all of the related articles and make their own determinations.

    1. Emily,

      Thanks for correcting my name.

      When you called me out on Twitter I responded to let you know that I was rewriting my story. This was thanks to an offer from Alternet to actually pay me, unlike at The Atlantic, and give me input in the headline. I probably did refer to your initial response in my re-write, as I took input from others as well, both negative and positive—including helpful input from a former Monsanto data guy, the editor of a high-profile science magazine, a genome statistician, and others, many conducted privately via email because few of my supporters were inclined to appear at all supportive of me in public (I can’t imagine why). While the re-write wasn’t perfect, it was the stronger. And I an only assume, by your obvious pride at mistakenly taking credit for the re-write, that you were satisfied with your handiwork. But then, why the attitude?

      To add to your above list of pieces you’ve written about me and my article,I seem to remember a fourth that you worked on. In response to your request for comment for this fourth story I had reminded you that I had clearly stated: “Should the research survive scientific scrutiny — a serious hurdle — it could prove a game changer in many fields.” I remained dumbfounded that, even after making this point (in both drafts), you kept shitting on me.

      For what it’s worth, the way in which you characterized one of my “leaps” is actually fair to me. Despite your dramatic use of “leaped,” I generally agree with your packaging of one of my points into an “if-then” statement:

      “The food writer who picked up on this research, Ari Levaux [sic], leaped from this initial finding to the conclusion that if regular rice RNA could do this, then so could RNAs from GM foods.”

      One is allowed, of course, to make “if-then” statements, in both science and journalism. This particular “if-then” statement was importanct, as you and I both pointed out, because many 2nd and 3rd-generation GM constructs employ miRNA. Thus, the activity reported by Zhang, if valid, could be a very big deal, not only for GM food but also in therapeutic applications, as I mentioned in both my Atlantic draft and the re-write.

      So if your point, in your most recent response to my work, was that you believe to have verified that my “if-then” was indeed a dead end (and the jury, many would argue, is still out on this), I think that is a perfectly worthwhile ambition. More power to you. Buy why the patronizing attitude? Why should I get more ink, in your Forbes story, than either Zhang or Witwer? And why take me out to the woodshed four times about the same article that you admit to being intrigued by?

      “…the story proved so interesting to me that I pursued it further…”

      How many more articles I should expect from you, until this obsession of yours runs its course?

  14. @Susan Fairbairn (nee Kirk): You think rice should have been kept in the lab? Wow. Good thing early farmers didn’t have your advice. Lots of people around the world have relied on rice for major parts of their diet for…well…centuries.

  15. Emily, My apologies, FWIW. I see the idea of you and Wilcox taking credit for the re-write came from Virginia Hughes’ column, above, when she wrote:

    “*That column was rightfully struck down by science bloggers Emily Willingham and Christie Wilcox, and because of their posts, the author eventually amended it. The self-correction of the blogosphere is just a tad faster than the self-correction of science, eh?”

  16. I disagree that there is much right in Christine Wilcox’s blog. She did not understand the biology. She writes:

    First off, there’s no reason to think that the DNA being introduced into GMOs is going to produce more/different miRNAs than it did in the original organism. Ari’s claim that “new DNA can have dangerous implications far beyond the products it codes for” simply isn’t true because miRNAs are coded for. These small RNA fragments aren’t random or accidental – they are explicitly detailed within the genome.

    This is fundamentally incorrect. Small RNA fragments can arise via either the miRNA pathway or via the siRNA pathway. Once the longer RNA is cleaved into small RNA the miRNA and siRNA pathway largely converge, as described in comment 36 under her blog. Once they converge they function similarly. Therefore, an siRNA would have the same effect as a miRNA, and siRNAs are accidentally introduced into GM food, and by law not required to be tested for. Failing to address the fact that these two pathways converge was also a factual error that Emily Williamson also made in her Slate article. Monsanto scientists, however, correctly and to their credit tacitly acknowledged the overlap in their press release.

    This is critical to understanding the initial controversy because the idea that small RNAs could be accidentally introduced into a GM product is not hypothetical. It will occur whenever any double stranded RNA is created. This happened in 1994 when scientists introduced an antisense RNA into the FlavRSavr tomato and turned on the siRNA pathway. This was not known until a decade later, well after the food was on the market.

    I don’t think it adds to the discussion to have biologists commenting on issues as experts when they do not understand the pathways that they are writing about. It is laziness disguised as sanctimony.

    I thought Dr. Zhang’s response was excellent, especially his description of the biases introduced into the sequencing process by inherent differences in animal and plant small RNAs. Whatever its implications for GM food, it is interesting science and I hope we will hear more about it.

  17. Science in no way encompasses the “Mystical.” Science is man made. The study of the unknown. If god had started science then the mystical would be a relevant term and consequently …all mysteries would be known therefore reducing the need for scientific energy. Man takes time to ‘anything’ worth knowing.

    And for all the impatient who feel that science is glorifying the “Self-correcting” notion and using it as a catchall –I told you so– my suggestion is to sit down with your god and ask if there is a better way!

  18. I have questions about the microRNA, could it be possible that humans are producing the RNA to help regulate in certain metabolic processes since you mentioned how a study of certain althetes showed no miRNA which decreases the uptake of cholesterol which would make sense because athletes burn through a large amount of calories and such wouldn’t requre such a molecule. Also when the tests were conducted using PCR was the same segment copied or were different labs copying different sections of RNA this could cause discrepancies especially if another form of RNA in the blood had the same short sequence although wasn’t from the plants?

  19. Considering the great issues discussed here I feel somewhat abashed at the insignificance of my observation, but as an allergy sufferer I find it quite odd that people are surprised at the occurrence of non-human RNA and other substances within the human body. Plenty of genetic material is inhaled, absorbed through the skin, swallowed, that bypasses mainstream digestion. Some of it is doubtless ignored by the immune system and other filtering mechanisms. Some of it isn’t, as I am reminded every September.

  20. How many studies showing mice developing tumors from eatng GMO’s does it take before science self-corrects on the “GMO’s are safe” issue?

Leave a Reply

Your email address will not be published. Required fields are marked *