National Geographic

Arsenic bacteria – a post-mortem, a review, and some navel-gazing

Mono_Lake

It was the big news that wasn’t. Hyperbolic claims about the possible discovery of alien life, or a second branch of life on Earth, turned out to be nothing more than bacteria that can thrive on arsenic, using it in place of phosphorus in their DNA and other molecules. But after the initial layers of hype were peeled away, even this extraordinary claim started falling under suspicious glances.

I’ve already talked about the untrammelled hype that surrounded this paper and I won’t retread that ground again. This is a chronological roundup of the criticism against the science in the paper itself, ending with some personal reflections on my own handling of the story (skip to Friday, December 10th for that bit).

Thursday, December 2nd: Felisa Wolfe-Simon published a paper in Science, claiming to have found bacteria in California’s Mono Lake that can grow using arsenic instead of phosphorus. Given that phosphorus is meant to be one of six irreplaceable elements, this would have been a big deal, not least because the bacteria apparently used arsenic to build the backbones of their DNA molecules.

In my post, I mentioned some caveats. Wolfe-Simon isolated the arsenic-loving strain, known as GFAJ-1, by growing Mono Lake bacteria in ever-increasing concentrations of arsenic while diluting out the phosphorus. It is possible that the bacteria’s arsenic molecules were an adaptation to the harsh environments within the experiment, rather than Mono Lake itself. More importantly, there were still detectable levels of phosphorus left in the cells at the end of the experiment, although Wolfe-Simon claimed that the bacteria shouldn’t have been able to grow on such small amounts.

Friday, December 3rd: Various commenters voiced their concerns, and I attached an update to my post with a comment from John Sutherland, a skeptical biochemist in Cambridge. He argued that the arsenic-based DNA wouldn’t have been stable enough in water, and said that the authors need to actually create some arsenic-based DNA to study in detail.

Saturday, December 4th: The backlash began in earnest with several scientists criticising the conclusion that the bacteria have actually substituted arsenic for phosphorus in their DNA. Microbiologist Rosie Redfield posted a scathing critique on her blog, pointing out several flaws in heavy technical detail and dismissing the paper as “lots of flim-flam, but very little reliable information”. Among several objections, she noted that the bacteria grown in arsenic were still surrounded by more than enough phosphorus for their needs. The DNA was never purified and instead analysed while embedded within a gel; as a result, the tests could have picked up arsenic stuck around the DNA or loitering within the gel.

Alex Bradley, a geochemist and microbiologist, posted his own lengthy analysis, noting that the bacteria’s DNA was immersed in water, which should have quickly broken any fragile arsenic-based compounds. It’s more likely that the DNA had a typical spine of phosphorus. Bradley also says that bacteria in the Sargasso Sea can indeed grow on the very low concentrations of phosphorus found in the GFAJ-1 cultures, contrary to the claims of Wolfe-Simon’s team. The technique that could have resolved the debate – mass spectrometry, which analyses the elements that make up a molecule – wasn’t done. “Unfortunately,” Bradley said, “these exciting claims are very, very shaky.”

Sunday, December 5th: The backlash builds. Several posts make mention of Redfield’s and Bradley’s critiques, and David Dobbs carefully considered whether science journalists could have done a better job with the story. “We thought we were getting cupcakes. Some of us wanted cupcakes. Who doesn’t want cupcakes? Now, everybody’s got humble pie in front of them, quite a bit to eat yet, and no dessert on the menu.”

Monday, December 6th: The mainstream media started to pick up on the backlash. CBC News (I think) were first out of the stables, with strong quotes from Redfield: “I blog openly…to bring this stuff more into the open where everybody can see it.”

Meanwhile, the first signs emerged that NASA weren’t going to engage with the criticisms. Dwayne Brown, their senior public affairs officer, highlighted the fact that the paper was published in one of the “most prestigious scientific journals” and deemed it inappropriate to debate the science using the same media and bloggers who they relied on for press coverage of the science. Wolfe-Simon herself tweeted that “discussion about scientific details MUST be within a scientific venue so that we can come back to the public with a unified understanding.”

Tuesday, December 7th: It got worse. Carl Zimmer, writing in Slate, called up a dozen scientists, the vast majority of whom said that Wolfe-Simon hasn’t made her case. One of them frankly stated, “This paper should not have been published.”

NASA’s intransigence continued. In a frankly astonishing quote to Ivan Oransky, Brown asserts, “NASA DID NOT HYPE anything – others did. Credible media organizations have not questioned NASA about any text. Bloggers and social media have.” Wolfe-Simon herself tells Zimmer that media debates “do not represent the proper way to engage in a scientific discourse and we will not respond in this manner.”

It didn’t go down well. Jonathan Eisen says that “they carried out science by press release and press conference” and “are now hypocritical if they say that the only response should be in the scientific literature.” David Dobbs calls the attitude “a return to pre-Enlightenment thinking”, and rightly noted that “Rosie Redfield is a peer, and her blog is peer review”.

Chris Rowan agreed, saying that what happens after publication is what he considers to be “real peer review”. Rowan said, “The pre-publication stuff is just a quality filter, a check that the paper is not obviously wrong – and an imperfect filter at that. The real test is what happens in the months and years after publication.”Grant Jacobs and others post similar thoughts, while Nature and the Columbia Journalism Review both cover the fracas.

Nonetheless, later in the day, NASA arranged for a quirky lecture about the findings. After some bizarre goofing-off, Oremland addressed a few of the criticisms. He said that lack of money prevented them from doing mass spectrometry experiments. And contrary to Bradley’s point about Sargasso Sea bacteria, he reiterated the claims in the paper that GFAJ-1 couldn’t have grown on the “smidgen” of phosphorus it contained. He finished by encouraging other groups to try and replicate their experiments, even offering to send them the bacteria.

Meanwhile, Eisen urged critics to “make this an open discussion of science and science reporting and not a venue to spout derogatory comments about the people involved.”

Wednesday, December 8th: Wolfe-Simon posted a public statement on her website. While she welcomed the “lively debate”, she didn’t address any criticisms. Promising an FAQ about the paper later, she invited others to “read the paper and submit any responses to Science for review so that we can officially respond”. She’ll get her wish – Redfield has already formalised her critique in a letter to Science.

Wolfe-Simon’s viewpoint garnered some support – Jack Gilbert at the University of Chicago said that impatient though he is, peer-reviewed journals are the proper forum for criticism. Others were not so kind. At the Guardian, Martin Robbins says that “at almost every stage of this story the actors involved were collapsing under the weight of their own slavish obedience to a fundamentally broken… well… ‘system’” And Ivan Oransky noted that NASA failed to follow its own code of conduct when announcing the study.

Carl Zimmer, ever the journalistic innovator, posted the full text of all his Slate interviews on his blog. The responses are illuminating. Many of the points had been made but notably, Hazel Barton (another microbiologist) was one of the few figures to publicly criticise Rosie Redfield’s review. She said, “It’s important that the reviews of individuals who are not experts in the field do not have as much weight as that of the original reviewers, no matter how public the review. I felt that her review had an equal number of flaws in it.” She praised Alex Bradley’s critique instead, as did his former boss Roger Summons.

The list of mainstream outlets to cover the controversy grew to include Wired, New Scientist, the Daily Mail, the Atlantic, and many more. Much of it went over well-trodden ground, although Carmen Drahl added something new to the mix with more detail about the chemical analyses used in the study. Meanwhile, Science made the article freely available to the public for the following two weeks.

Thursday, December 9th: It seemed that most of the criticisms had been voiced but some scientists spoke out in support of Wolfe-Simon’s reaction. Dr Isis said, “If question remains about the voracity of these authors findings, then the only thing that is going to answer that doubt is data.  Data cannot be generated by blog discussion… Talking about digging a ditch never got it dug.” Doctor Zen wondered if people only want Wolfe-Simon to engage with her critics to see a public smackdown: “I almost get the sense that they think that debating science bloggers would be like wrestling with a pig: they’ll get dirty and the pig would enjoy it.”

Friday, December 10th: And that brings us up to now. Looking back, it is astonishing how quickly these events unfolded and the sheer number of bloggers and media outlets that became involved in the criticism. This is indeed a brave new world, and one in which we are all the infamous Third Reviewer.

Others have discussed these issues in greater depth and the links above should provide a good cross-section of opinion. For my part, I wanted to think about my own handling of the story, especially because I’ve been criticised on Twitter for dropping the ball on it. I don’t actually disagree. This is what I tweeted on Sunday:

Arsenic_tweetTo clarify that, I tried to quell the hype around the study as best I could. I had the paper and I think that what I wrote was a fair representation of it. But, of course, that’s not necessarily enough. I’ve argued before that journalists should not be merely messengers – we should make the best possible efforts to cut through what’s being said in an attempt to uncover what’s actually true. Arguably, that didn’t happen although to clarify, I am not saying that the paper is rubbish or untrue. Despite the criticisms, I want to see the authors respond in a thorough way or to see another lab attempt replicate the experiments before jumping to conclusions.

However, the sheer amount of negative comment indicates that I could have been more critical of the paper in my piece. Others have been supportive in suggesting that this was more egg on the face of the peer reviewers and indeed, several practicing scientists took the findings on face value, speculating about everything from the implications for chemotherapy to whether the bacteria have special viruses. The counter-argument, which I have no good retort to, is that peer review is no guarantee of quality, and that writers should be able to see through the fog of whatever topic they write about.

There is no easy answer to this. On Twitter, my response was that we should expect people to make reasonable efforts to uncover truth and be skeptical, while appreciating that people can and will make mistakes.

So for me, it comes down to this: did I do enough? I was certainly cautious. I said that “there is room for doubt” and I brought up the fact that the arsenic-loving bacteria still contain measurable levels of phosphorus. But I didn’t run the paper past other sources for comment, which I typically do it for stories that contain extraordinary claims. There was certainly plenty of time to do so here and while there were various reasons that I didn’t, the bottom line is that I could have done more. That doesn’t always help, of course, but it was an important missed step. A lesson for next time.

To conclude, my aim in this is not to flagellate myself in a dervish of self-indulgence. However, I do believe that it you’re going to try to hold your profession to a higher standard, you have to be honest and open when you’ve made mistakes yourself. I also think that if you cover a story that turns out to be a bit dodgy, you have a certain responsibility in covering the follow-up. Hence, this post.

Image by Michael Gabler

If the citation link isn’t working, read why here

tweetmeme_source = ‘DiscoverMag';
tweetmeme_service = ‘bit.ly';

Twitter.jpg Facebook.jpg Feed.jpg Book.jpg

There are 58 Comments. Add Yours.

  1. Janne
    December 10, 2010

    A basic problem with is the embargo. Specifically that journalists get early access, while peers – other specialists in the field – do not. It means that the journalist, like yourself, can rely only on the original authors, with no way of getting other views on the findings. And it means that peers can’t write about the paper when the journalists (who, inevitably, do a positive-only coverage due to the lack of other viewpoints) do, but will be able to voice only after they’ve been able to digest the paper and formulate a response.

    Best solution: abolish embargoes. Everybody gets access at the same time. If a news organization won’t cover a finding without preferential access – well, tough. It’s not like any particular outlet is exclusive any more.

    Second best (distant second): widen early access to include not just journalists but a selection of peers. Both to get their responses on blogs and similar, but also to give journalists access to other informed opinions when writing their pieces.

  2. Walter S. Andriuzzi
    December 10, 2010

    RIP Arsenium lovers

  3. Ed Yong
    December 10, 2010

    It means that the journalist, like yourself, can rely only on the original authors, with no way of getting other views on the findings.

    No, that’s not true. The embargo doens’t preclude journalists from sending papers out to other authors for review and comment. I do this a lot and I have been critical about new papers as a result, but that’s the step that I missed for this story.

  4. Bill Gleason
    December 10, 2010

    My initial reaction was published on the Brainstorm blog at the Chronicle site:

    Hic, Haec, Hype – Arsenic in DNA or Little Green Men
    Link: http://bit.ly/eKOwO1

    As a chemist, my problem was with the stability of the DNA. However, there are possible explanations for this, perhaps farfetched, but…

    I think it was a pretty even-handed piece and for a couple of days I updated it by adding references to people like Ed and Rosie.

    In the end, as has been pointed out, the answer will have to be provided in the lab:

    Is there arsenic in the DNA of these bugs?

    Pointing out the experimental flaws in this work will not answer that question.

    The other thing that amuses me is that the quality of work reported in Science/Nature is NOT always the highest. I was amused to learn recently that the quality of X-ray crystal structures is MUCH higher than those found in Science and Nature in the less prestigious, but still very good ACS journal, Biochemistry.

    You did a great job, Ed. Trying to keep up on this on a day to day basis must be tough indeed.

    As politicians say: Mistakes were made.

    My best.

    Bill Gleason, University of Minnesota

  5. Ben
    December 10, 2010

    Interesting post mortem. I’m rather concerned, hwoever, that Doctor Zen’s “pig wrestling” comment is exaclty the way many scientists will see it — and thereby be discouraged from participating in this Brave New World.

    A lot of the online coverage of this story has been positively gleeful in its muck-raking. I admit I felt somewhat the same way when the story began (for me, Paul Davies’ speculation in the media seemed entirely unnecessary, but beyond that i’m no expert in the area). But no scientist wants to be on the receiving end of what has happened in the last several days. And, I don’t know that any scientist can be entirely sure when they might encounter some measure of the same thing, if they are going to engage with the media. That is doubly true for early-career scientists like Dr Wolfe-Simon (and, incidentally, myself).

    It’s clear, of course, that many of the legitimate criticisms (as opposed to the various ad hominem attacks) will carry a lot of weight in whatever science proceeds on this topic. From a media standpoint, it’s also obvious that NASA and Science, and the authors of the paper, all made big, public errors in theway they handled the coverage of their work. But rather than being a triumph for open science, I can’t help but feel that almost everyone has lost something here.

    Science bloggers and anyone else with an interest in more open scientific discourse should be reaching out to scientists who don’t yet share their vision of a better future, not giving them reasons to fear it.

    Ben (@tinscience)

  6. Ed Yong
    December 10, 2010

    A great point, Ben. That’s exactly why I linked to Jonathan Eisen’s plea for people to champion the open discussion of science without turning it into a personal attack on the scientists themselves.

  7. Elsa
    December 10, 2010

    Each time this happens–the swarm of positive coverage for a study, followed by the devastating critiques–I wonder why the journalism didn’t work better. Did journos contact the “wrong” outside experts for comment for the first wave of stories? Was there not enough time for the experts to do a critical reading before the embargo lifted? Of course most of the first stories predictably reflected some reservations, but why didn’t the “shouldn’t-have-been-published” quotes come til later? I have some scientist-friends who, when contacted by journos for comment, hesitate to provide negative feedback on a story, preferring to just remain silent. Hopefully events such as these will help them understand why it’s good to speak up and help the chances of getting level-headed coverage from the start.

  8. Janne
    December 10, 2010

    No, that’s not true. The embargo doens’t preclude journalists from sending papers out to other authors for review and comment. I do this a lot and I have been critical about new papers as a result, but that’s the step that I missed for this story.

    I didn’t know that, and I stand corrected. That is good to know.

    However, I do wonder a bit how good the uptake is on this. I mean, if I suddenly got an email from a journalist saying that “here’s an upcoming paper I’m writing about and I’d like your comments on it” then it would probably drop a fair bit down on my priority list since I can’t really do a comment on it on my own blog. What is the response-rate in general for you on this kind of pre-publication comments?

    Also, would you say that the most relevant critics of this paper so far would have been people that would have been contacted before publication by journalists? I mean, it certainly doesn’t seem like anybody in the media have talked to anybody at all with a critical take on the paper. We’re all prone to bias in our work, and I would not want to trust a journalist to deliberately and reliably pick second-commenters that could potentially kill their story. I know I wouldn’t.

  9. Don Monroe
    December 10, 2010

    Thanks for your self-criticism. I suspect that part of the problem was that you and others were justifiably proud that you didn’t get taken in by the extraterrestrial diversion, and so were lulled into relative complacency on the more earthly issues.

  10. Ed Yong
    December 10, 2010

    @Elsa – This is true. As David Dobbs wrote in his post,

    “But as the experts usually lack time to compare impressions with peers, few will go out on a limb and really lay into a paper under those circumstances. You usually get either “This looks interesting, with a few caveats I’d like to note” or “I’d rather not comment.” You’ll rarely get an outright dismissal. They lack the time and probably the taste for the trouble it’ll make.

    @Janne – I get a pretty high response rate, in the order of around 80-90%, depending on the field, really. The problem with a story like this is who to contact. This is pretty interdisciplinary stuff – there’s the microbiology, the chemistry, the astrobiology angle. Ideally, you’d want to get a range of views on all those aspects, which is largely what Carl Zimmer eventually did.

    Also to pick up on your point about “killing the story”, if I got a bunch of comments that all said a paper was rubbish, that doesn’t necessarily mean I’m going to bin it. Showing that scientists disagree with each other (over legitimate debates!) is an important part of showing the public that science is a human endeavour and taking it out of the ivory tower.

    @Don – I suspect that you’re exactly right.

  11. Emily Willingham
    December 10, 2010

    I’m not clear on how the paper escaped the requirement to demonstrate arsenic in the DNA. Is that something anyone who is directly associated with the paper has addressed?

    The issues here are many. The science of the paper seems to be missing some pieces (see above). NASA handled this poorly beforehand, likely attracting the interest of those even with only a casual interest in science with overpromising, then having only “arsenic in bacteria from California” to offer. Those NASA promises of turning things upside down, making us think of life in a new way? Overstated nonsense, and people who may have been drawn to something science related were no doubt just as let down as scientists who grasp where the faults lie in both the presentation of the ideas and science of the paper.

    To assert that scientists who blog or bloggers with a deep understanding of science are not worth engagement is a dangerous stance that runs counter to the best (and possibly most idealistic) aspects of scientific discourse: an open airing and discussion of faults and fallacies and findings. The scientific blogosphere is, in its way, no different from a UK science men’s club of the 19th century with one powerful exception: We’re all welcome in this club. NASA and other entities that persist in their efforts to keep scientific discourse exclusive to journal-based exchanges will find those efforts wasted. Thought and discourse in science have broken the print/journal barrier, and they need to get used to that and learn how to deal with it. We are not pigs, and this isn’t dirty work. It’s relevant and probably the best thing that’s happened to scientific exchange in a long time.

  12. Gaythia
    December 10, 2010

    Ed, I believe that what you said here is vitally important:
    “Showing that scientists disagree with each other (over legitimate debates!) is an important part of showing the public that science is a human endeavour and taking it out of the ivory tower. ”
    For scientists, it needs to be possible to publish preliminary results, and, even if not fun, risk being wrong. Historically, papers were frequently reviewed by and then read by a fairly incestuous group of interlinked scientists. Turning this into a world wide free for all is a fruitful way to get to the bottom of things. But if it causes scientists to be too hesitant to come forward with ideas, and wait for perfection, then science loses.
    Another way for science to lose if it can’t defend itself against powerful economic forces that cause scientific results to be hyped or discredited in ways that are harmful to the process of gaining knowledge.
    All of which makes the role of science journalists a very significant one!

  13. Michele
    December 10, 2010

    Hi Ed,

    We’ve been debating about our summary of the article, wondering if we read it critically enough as well. I like the point that only more experiments and more data will settle the matter. I did write about my concerns about how all parties handled the communication of this information to the public at large though: A teachable moment in science communication (http://wp.me/pnsRh-264) . Maybe the most important thing we learn from this study is how to talk about science.

    Michele

  14. Pat Foster
    December 10, 2010

    “Showing that scientists disagree with each other (over legitimate debates!) is an important part of showing the public that science is a human endeavour and taking it out of the ivory tower. ”

    Except that it feeds into the perception by the public that science is a matter of opinion instead of demonstrable facts. This leads to the manufactured “controversies” about evolution, climate change, HIV, etc. Read Ben Goldacre on this subject

  15. Emily Willingham
    December 10, 2010

    @Pat…so what would be the solution? Conspire behind closed doors to present a united front? Some of the public already thinks that we conspire (climate change, vaccine/pharma etc.). There should be some ground-up solution for the general public, better training in critical thinking, acceptance of conflict over ideas, an understanding of the differences between facts and opinion, an emphasis on understanding the practice of scientific investigation, the evolution of ideas. Oh, of course there should be. But that’s probably not going to happen–certainly not in the current atmosphere. Which takes me back to my original question–what do you do about it? Closing off public discussion doesn’t seem to me to be the right approach, and neither does insisting on confining discussions to journal publications hidden behind paywalls. Viva la open discourse.

  16. Joe H.
    December 10, 2010

    There’s an aspect of this fracas that is full of real tragedy as well as real hope. It is that this paper and the surrounding coverage ignited an scientific interest in people who would normally never pay attention to such things. It was being posted left and right by non-scientists, grandmas, teens, a true outreach victory . . . story of the year type stuff. That’s the amazing victory of it all, and the fuel for the fire.

    The tragedy sets in when the criticisms revealed that the emperor was, in fact, half-naked. This paper may not eventually stand up to its criticisms, and that will be a sad day. Not for astrobiology or for microbiologists, but because it will have turned into a great hoax. People were finally paying attention, they were finally excited and we should all feel great about that. But if that excitement is based on hype and bad science in the end, there may be a great deficit of trust to overcome in the world of the non-scientist.

    I still think that getting the “general public” (still looking for a better term there) enthused and hyped is never all bad, but it walks a fine line when it comes to what stands behind the hype.

  17. Brian Romans
    December 10, 2010

    I really like Chris Rowan’s take on this (which you mention and link to). Peer review happens before submission (e.g., presenting prelim data at a conference, informal conversations, etc.) and after the paper comes out in addition to the ‘official’ review of the paper itself. Review doesn’t really have a beginning or an end, it’s always happening. And it’s inherently messy.

    As the review process becomes more accessible in ‘real time’, the observers — whether they’re journalists, other scientists, or the public — should appreciate both the nature and the timescale of the process. The expectation should be that it’s confusing and messy, not the other way around. The coverage of our modern world is dominated by minute-by-minute, hour-by-hour updates. This is *really* fast compared to the time it takes experts to read, digest, and synthesize their thoughts about their peer’s work.

    As a researcher who is active online, I do want to see true science — often messy, obscure, and confusing — conducted openly and accessible. When it’s cut and dry the story is about the science. When it’s not, which will be more often, the story will be about the process.

    It remains to be seen how many people will watch every play of the match, and how many just want to know the final score. [imperfect analogy, I know]

  18. David Dobbs
    December 10, 2010

    I’m very much with Brian Romans on this: It’s important to see — for everyone to see — that science, like democracy, is messy. Why shouldn’t it be? It’s people arguing about important ideas.

    Science is often messy, and often involves scientists arguing passionately about results, methods, conclusions, the whole shebang, as they are here. It’s valuable that everyone see this, especially in a case like this one that in the realm of things is fairly inconsequential from a political point of view. It’s a big deal to people in the field, and everyone is interested in the roots of life and ETs and so on. But compared to climate, the political stakes are low. So it helps that people see that this is messy — so that when they see the science behind things like climate is messy, they’ll understand that’s perfectly normal rather than some sign of foul play and subterfuge.

    If everyone engages in as much critical self-examination as Ed Yong does here, much good will come of this.

  19. Jim Handman
    December 10, 2010

    I must say I have been extremely puzzled and bothered by 2 tendencies that I’ve observed during this chaotic week of blogging and Tweeting about the arsenic controversy:

    a – the apparent malicious glee that many bloggers seem to take in pointing to the criticisms of the paper (and that includes some of Ed’s own Tweets). With few exceptions, everyone seemed to assume the authors were deliberately misleading, or had falsified results, or had intentionally disregarded certain observations, or were just plain sloppy scientists. Plus, there were some very nasty, inappropriate and personal attacks – including an incredibly sexist comment about Wolfe-Simon’s hair colour.

    b – the rush by virtually everyone on the web to side with Rosie Redfield over Wolfe-Simon and Ron Oremland (whom we interviewed on our program). Both Redfield and Oremland are highly respected senior scientists, and I have no reason to believe one over the other. I don’t understand why the vast majority of bloggers (most of whom are not microbiologists or organic chemists) instantly assumed Redfield was right. I’m just a simple journalist, compeltely unquliafied to judge the claims of either. So like Ed, I await scientific attempts to replicate the results, before I “rush to judgement”.

  20. Ed Yong
    December 10, 2010

    @Gaythia – On the topic of scientists being hesitant on the back of criticism, The Curious Wavefunction has a great post out today about exactly this, and some possible solutions.

    @Joe – I’m an idealist about this. It is no use getting the public to take a real interest in science if you’re going to feed them rubbish. It corrupts the ‘victory’ entirely. If the only way to engage people with science is to distort, you might as well not bother.

    @Pat – I strongly disagree. Science is indeed about the hunt for an objective fact-based reality; I have no truck with postmodernism. BUT, the road to that reality is rough and windy and littered with obstacles. Evidence is subject to interpretation and people layer their opinions over the facts. Hence, the debates. I very much agree with Emily, David, Brian and others that laying out this messy side to science is vitally important. If anything, outlining the actual controversies will help to show how farcical the manufactured ones are. This is what it looks like when real scientists actually disagree. As David said, “They’ll understand that’s perfectly normal rather than some sign of foul play and subterfuge.”

  21. Ed Yong
    December 10, 2010

    @Jim – I don’t think that I said anything personal or malicious towards the authors, or accused them of crap science? From recollection, my criticisms were over the refusal to engage with the growing debate. I did excitedly point to criticisms of the paper, because (a) with a journalist hat on, it’s an interesting story, and (b) with a scientist hat on, I *do* get excited when papers get criticised in a public forum. And it’s more than the bloodsport tendency that DoctorZen alluded to, it’s for the many reasons outlined in this comment thread, I think it’s a healthy thing. Also I did say something about the fact that people were automatically assuming that the paper was rubbish: http://twitter.com/edyong209/status/11585268108754944

  22. Chris Rowan
    December 10, 2010

    Ed, although I’m not going to argue with the lessons you’ve seen fit to draw for yourself from this, I personally feel you did a much better job than most in covering this – your initial coverage was appropriately measured, and you have covered the growing backlash in a timely and responsive manner. It’s easy to say in hindsight you should have dug more, but it wasn’t until NASA so firmly injected this thing into the public eye last Monday that digging into the claims of this particular paper took on such urgency.

    The problem only really arises if you consider a published paper to be a finished product. The fuss around this story seem to be driven by surprise that a paper can draw such criticism after making it through peer review as much as the criticism itself. But really it’s just a starting point for further discussions, and I can’t really blame you for not writing about those discussions before they actually happened.

    [NB: Tried posting this earlier from my phone but it didn't seem to take - I note than Brian and David have both made similar points in the interim]

  23. Ed Yong
    December 10, 2010

    @Chris – Thanks mate.

    In the meantime, Wolfe-Simon has retweeted me: http://twitter.com/ironlisa/statuses/13283760799744000 I can’t really make heads nor tails of the hashtags but there is possibly an implication that I didn’t read the original paper? For the record, I did. At length.

  24. Rosie Redfield
    December 10, 2010

    I don’t fault Ed or other science journalists at all. I think that they did the best they could, and that the various scientists contributing to this discussion have set a wonderful example of how science can be done on the internet.

    On the possibility of ‘gleeful muckraking’, I think I’m not the only scientist who was at least partly motivated by the entirely honourable pleasure of carefully analyzing data to see what it means.

    @Jim Handeman: My criticisms were presented as explicit descriptions of weaknesses I saw in the paper, not as “Believe me, I’m an expert”. Anyone with basic knowledge of molecular biology and/or microbiology could then look at the paper and make their own evaluation (provided they had access…). Based on the comments to my post and elsewhere, many people felt that my concerns were justified. And contrary to Hazel Barton’s implication, I limited my critical comments to issues on which I’m reasonably expert: bacterial growth, DNA purification, and simple data analysis. I didn’t critique the statistics or the elemental analysis because I don’t know enough to do that.

  25. Zen Faulkes
    December 10, 2010

    Jim Handman: “I don’t understand why the vast majority of bloggers… instantly assumed Redfield was right.”

    I don’t think people agreed with every aspect of the Redfield’s critique, but DID agree that the case was not proven. The entire reason that this study warranted the level of attention it did was that the claim being made was big, bold, audacious, and very cool… if it were true. And anyone who does science should know that if you want to make a big bold claim, it has to be airtight.

    I look forward to more coverage on Quirks!

    Ed: Thanks for the link, mate.

  26. Gaythia
    December 10, 2010

    Ed, I agree, the Curious Wavefunction post is great, but your link above is not working.

    [Fixed - Ed]

  27. Ben Cairns
    December 10, 2010

    @Rosie re: “gleeful muck-raking”, the large majority of the scientists responding to the paper online have been firm but appropriately measured in their comments. Rare exceptions are mentioned in the post and the above replies. Overall, the scientific response is great; Carl Zimmer’s posting of all those comments is something rather special. But there’s been a nasty undertone to some of the wider discussion, and it’s that to which I was referring with that phrase.

  28. Dagmar
    December 10, 2010

    There are several issues here that are easily conflated:

    1. Are the author’s conclusions correct? Those that are demanding data have the correct answer for this one. We won’t know until more data is out there.

    2. Did peer review fall down in this instance? The answer appears to be “yes” but without knowing what happened we can’t say for sure. Some journals (e.g. EMBO J) are publishing the editorial process along with each paper. If we had that information we could see what went wrong in this case, because something certainly did. Science and Nature can’t seem to learn their lesson as they get caught up with these big hype papers.

    3. Are the authors of the paper culpable given that their paper was peer reviewed and they could turn out to be right? I think they are. A fundamental question for any paper is whether or not the conclusions follow from the data. In my opinion this paper falls woefully short in this regard. Even if it turns out that their conclusions are correct, this paper is still very poor – basically the authors would be getting lucky. The bottom line is that the authors submitted a substandard manuscript and it made it through peer review. Apparently the authors are now trying to say this is all part of the scientific process, which is true but doesn’t change the fact that they submitted sloppy science and must be held accountable.

  29. Krista
    December 10, 2010

    This whole debate is very interesting. What I’d like to know is how this paper, which seems to lack conclusive data, was published in Science in the first place. I think the journal itself may be getting off too lightly. The chain of responsibility, in my mind, begins (after the researchers themselves) with the peer reviewers and Science editors who decided to publish the paper in the first place. Next in line would be the press office people at Science and NASA. Finally, journalists and bloggers who may have been less critical than they should have been.

    This has been a learning experience for me, regardless. I am a public information officer at a major US medical school. In our office we frequently discuss how to get the word out about interesting research without overhyping the findings. Sure, we can’t claim to be unbiased journalists, but it’s a line we’re very aware of and we take our jobs seriously. Sometimes we find ourselves managing the unrealistic expectations or claims of our own faculty members if we feel the research doesn’t warrant such attention.

    But even though I have a PhD in cancer biology and I read each paper thoroughly, I’m not an expert. And we do base our judgment about media coverage in part on the journal in which the research will be published; I’m more likely to take a paper seriously if it’s in Nature, Science or another top-tier journal than if it’s in a less-well-known publication (although we try to always write about well-done, interesting research regardless of where it’s published). So it’s disheartening to me to see first that a paper seemingly lacking essential data (mass spec) was published, and second that NASA would show such poor judgment in promoting it like they did. Suggesting it’s associated in our search for extraterrestrial life? Come on. That’s a slap in the face to PIOs who try to do their jobs thoughtfully and responsibly.

  30. Brian Romans
    December 10, 2010

    Krista says: “I’m more likely to take a paper seriously if it’s in Nature, Science or another top-tier journal than if it’s in a less-well-known publication”

    I disagree. To get a paper in Nature/Science you need high-/broad-impact over everything else. Being in Nature/Science doesn’t guarantee it’s correct, it guarantees that it’s exciting, ground-breaking, etc. (or potentially those things).

    The >99% of the rest of science is conducted in these “less-well-known publications” and much of it is accurate and of high quality. It just may be a very narrow study within a sub-discipline. Such studies can be *extremely* important for a discipline. Look at the reference list of a paper in Nature/Science, they all aren’t in the flashy, media-loving journals. People ought to take these papers seriously as well.

  31. Dagmar
    December 10, 2010

    There’s room for “gleeful muckraking” in Science. I disagree with anyone saying that “Science” has to take place in a particular venue. Criticism and skepticism, key components of scientific method, should be able to come from anywhere. The point isn’t where they come from, but their validity. Shame on the authors for refusing to address criticisms outside of the gated journal community. I can tell you that if this was one of my papers I’d be taking on all comers .

  32. brian (byers)
    December 10, 2010

    It seems this NASA event was more of an experiment in the use of embargo in the shortcutting of “normal” peer review. Another example of embargo screwup is available at Grist’s http://www.grist.org/article/food-2010-12-10-leaked-documents-show-epa-allowed-bee-toxic-pesticide-.
    I did not find the arsenic article as interesting even with it questionable aspects for zenobiologists os the uraniumphage trained to nano up platinum sponge. What is really interesting about the Arsenic article is the evolution of the misfire that led to that piece of show biz by Nasa (speaking of Rocket Science).

  33. Jada
    December 10, 2010

    I’m a little disappointed that most people have stopped talking about the science and moved on to talking about whether and how we should be talking about the science.

  34. Andrew Alden
    December 10, 2010

    This kind of controversy is very common in medical science. This biogeochemistry flap is about average compared to what you’ll see any week in medical news, and in my opinion it’s largely the fault of the mass media feeding the public a steady diet of fear, hope, sex and amazement. And compared to the heat of emotion about political stories, this flap is downright tepid. Scientists have to catch up to what kids already know: the world is all one big streetcorner now where everyone can speak up. Old-school journalists started out horrified that readers could actually argue back; since then I think they’ve raised their game. Now scientists are finding out the same things.

  35. Dagmar
    December 10, 2010

    @Jada: If you’re disappointed that people aren’t talking about science then why post a comment devoid of scientific commentary? Anyway, what’s wrong with discussing the scientific process?

  36. Ed Yong
    December 10, 2010

    @Jada – Why? The science is far from the only interesting thing about this story. It has big implications for peer review (pre- and post-), the way science is reported, the embargo process, the way scientists engage with the media, and more. And besides, I discuss a lot of the scientific content of the backlash in my round-up.

    @Andrew – I agree. I work in public health during the day and read a lot of epidemiological papers. These issues are par for the course. I think that old-school journalists are *starting* to raise their game but I don’t think that game-raising has become systemic through the profession yet! ;-)

  37. Shecky R.
    December 10, 2010

    Thanks Ed for this great succinct compendium…
    My one concern in all this is that controversial, hyped findings are pretty routinely the subject of intense critiquing and “smackdowns.” More routine (or popular) science/findings deserve JUST AS much critical scrutiny and questioning… yet often get a much freer ride, despite flaws.

  38. Trond Engen
    December 10, 2010

    If it’s true that the study is lacking in conclusive evidence for its claims, what’s NASA’s role in this? Financing the study, cutting the money at a time when the results looked spectacular but hadn’t been thoroughly tested, pushing for publication and hyping the article?

    In the movie the young scientist will find herself torn between her justified doubts in the validity of her conclusions and the pressure from her sleazy mentor, the dean of her underfinanced faculty and a father who she knows wants her to fulfill the dreams he had to give up when that terrible lab accident claimed his eyes all those years ago, but through the heated debate turned dramatic escape from government agents turned love scene with her harshest critic she finds the power to fight back just in time to save life as we know it.

    But in real life this is probably about honest people trying to fulfill their commitments — money-constrained scientists and publicity-nurtured space agency officials on one side, time-constrained reviewers and ad-nurtured editors on another — all hoping that this is what it seems to be and talking eachother up. Questions to ask oneself for players on all levels. When I started on this comment I was annoyed with NASA for pushing it through, now I wonder why nobody close to the scientists voiced concern with the process.

  39. alice
    December 11, 2010

    You seen this tool? http://www.dipity.com/

    Has anyone done one for this story? Would it even be useful? It’d probably miss the more context-filled and reflective nature of this post. Wonder if it’s worth using more as part of science storytelling through.

  40. AcademicLurker
    December 11, 2010

    Eugenie Reich, who wrote a book about the Schon scandal, gave a talk at my university this Spring. She had somehow gained access to the details of peer review for several of Schon’s Nature/Science papers (text of original reviews, editorial responses & etc.).

    In at least 2 cases, reviewers said essentially “This is a potentially very interesting result, but it’s not adequately supported by the data provided” and requested some additional experiments. In these cases, the editors decided to ignore the requests and publish anyway.

    Contra Krista at #29, I take exactly the opposite view. I think Nature/Science papers are less likely to hold water in the end than papers in the higher level field specific or society journals.

  41. Dagmar
    December 11, 2010

    @AcademicLurker: Editors at all levels sometimes have to rebuff reviewer requests. Reviewers aren’t always reasonable and authors have a chance to rebut their comments. Editors rightfully have significant leeway in deciding what that authors should be required to do before publication. Taking away this leeway would be a very bad thing. Of course sometimes mistakes can be made and it seems that heavily hyped results make this more likely to happen as the editor “wants to believe” as publication could further their career.

  42. AcademicLurker
    December 11, 2010

    @Dagmar: I agree that sometimes it can be the role of the editor to protect the authors from unreasonable requests for more experiments. In some areas of cell biology it seems that the requests for more data would continue indefinitely sometimes unless the editor calls a halt.

    My impression, though, is that in the specific cases of Nature/Science, the calculus is more along the lines of weighing the “splash” that a high flying claim will make against the probability that it’s inadequately backed up, and placing much greater weight on the former.

  43. Ed Yong
    December 11, 2010

    An alternative hypothesis: the perception that peer review is weaker in Nature and Science is due to confirmation bias. Papers in these journals, precisely because they are more likely to make a big splash, are more thoroughly critiqued by a wider range of scientists working across disciplines. As a result, their flaws are more likely to (a) be uncovered and (b) be uncovered in a very public way.

    Obviously, I have no data to support this but just throwing that out there.

  44. Sarah Kavassalis
    December 11, 2010

    I can’t figure out why people are criticizing the authors for publishing their findings in a high-impact journal, as is. If you write a paper, and think it has a shot at Science or Nature, you send it in and hope for the best. That doesn’t mean you think you’ve written the end-all-be-all on the subject, just that it might appeal to their readership (and score you some nice impact points). The fact that there are some experts who thinks the results warrant serious attention should probably be enough for it to be published exactly where it is. Debate is good for science; publishing contentious papers yields debate. Sure, the authors are confident, and some people don’t like that, but so are the authors of a great many papers that turn out to be incomplete/incorrect after further review. Very few ‘first studies’ on a topic turn out to be perfect, but that doesn’t mean that they shouldn’t published.

    If you’ve written something that sounds right to reviewers, even though it isn’t, that’s important, because it clearly shows something about the knowledge base/logic/thought process of the reviewers – think of the of the Alan Sokal hoax.

    I’ve really enjoyed how the scientific blogging community has handled this – especially Ed’s coverage – because it’s shown the entire spectrum of responses that you’d find in any scientific debate (people agree because they believe it, people agree because they blindly agree with authority, people disagree because they don’t believe it, people disagree because they have to disagree with anything hyped up to be a “paradigm” (sorry) shift, etc.). The only people that I think handled this poorly was NASA, for their silly hype and then outdated response to not wanting to discuss science in the media, blah, blah, blah, and the media outlets, who played up the hype and reported it as is (like that painful CNN interview Wolfe-Simons had to do), which is sadly what they do for all science coverage, so this no different than the norm.

    People are upset because Wolfe-Simons didn’t want to respond in blogs/online and preferred to have responses sent to Science? I completely get both sides to this, honestly. Scientists who are active on the internet love it and can’t imagine going back to more old school correspondence, but internet!scienctists still aren’t the norm in most fields. A lot of people completely stand by the old fashion vetting method of correspondence through journals, and yeah, it’s slower, but it’s a system that’s worked for a very long time, so it’s one people trust. And frankly, just imagine the incredible flood of emails these poor authors must have been receiving from people asking for responses to critiques. Especially since most of those emails are probably from non-scientists, ranging from pure crack-pots (who should probably be ignored) to interested outsiders (who’s emails can be fun to respond to the first few times, but imagine getting 100s of the same questions within days). I can not blame anyone, with press like this, for wanting a journal to screen their email, basically, because there would be no way of knowing, without opening and reading possibly 1000s of responses, which critiques were actually peer critiques and which were from people suggesting that it was a government conspiracy covering up the real truth of blah, blah, blah. If you’re not up with the science blogging culture, you could be entirely unaware that actual scientists do research blogging these days.

    The main things I hope that people take from this is that science coverage in the mainstream media can really suck (which shouldn’t be a surprise to anyone). Hopefully, the poor job in critically reviewing matters of science will get people thinking about other forms of media (like research blogging) that could have the possibility (but not guarantee) of providing a more critical analysis. Unfortunately, it’s really only the people who are already aware of the wider internet science culture that will probably notice this, so there won’t likely be many converts to more open source/open access/open comment way of discussing science… but, I can be hopeful.

    And come on, who here doesn’t find themselves saying that science needs more PR *all the time*. I may be very anti hype, but when it gets *millions* of people discussing science and (better still) the scientific process, I’d say that’s a win! A scientific discovery and criticism of that discovery on the news in the same week! How awesome is that? Well done, Wolfe-Simon, et al.!

  45. Bas Jansen
    December 12, 2010

    In all fairness, it is impossible to expect the authors at NASA to weed through everything that has been said on the Web, and then respond to it. In that view I can understand Wolfe-Simon.
    Furthermore, no one seems to realize that scientist these days have very little time for their graduate students, let alone for all the hidden piglets on the Web…

  46. Ed Yong
    December 12, 2010

    @ Sarah – fantastic comment. I loved the thoughtful, empathetic nature of it and the point about internet culture still being fairly niche is well taken.

    To clarify my own position, I’m not saying that Wolfe-Simon should be out on every single blog post answering criticisms, or be responding to all her undoubtedly numerous emails. However, I do think that there should be a more detailed response than what has been thus far provided, without having to wait for a formal journal process. There are several specific points of criticism that have been made that have been summarised in various articles. Addressing those in one fell swoop would be a good idea. As others have said, you can’t have your cake and eat it, getting all the benefits of widespread media coverage without addressing the subsequent backlash, even if it’s in the form of an isolated statement.

  47. Brian Romans
    December 12, 2010

    @EdYong re your idea that papers that make a bigger splash (i.e., get more coverage) are more thoroughly critiqued by peers, I’d agree — although I don’t have any data to back that up either. And perhaps not so much more thoroughly critiqued, but sufficiently critiqued by A LOT more peers than usual.

    As for peer review online, it seems that a model that would work in the near future — as people become more accustomed to interacting online — is for journals to have comment/reply “pages” set up for a paper that go up when the paper comes out. It seems there has to be some central place to do this. If I were an author and there were 10-20 blog posts out there it would be very difficult to find them all and then try to manage each and every thread. I don’t know, just thinking out loud … I’m sure there are folks who have thought a lot more about how to do this effectively.

    Awesome thread.

  48. Rosie Redfield
    December 12, 2010

    @Brian: Quite a few journals do have Comments pages for their papers, but we researchers don’t like to use them. I wrote a blog post pushing an alternative to this yesterday – click on my name to see it.

    [A direct link to Rosie's post here - Ed]

  49. Brian Romans
    December 12, 2010

    @Rosie Cool, I will comment directly on your post about your suggestion/alternative

  50. Extrasolar
    December 13, 2010

    Actually, what I hope this does is demonstrate to the public that the process of scientific peer review works! A paper might be published, but it still has to stand up to further scrutiny, and any errors (and truths) eventual come out in the wash. Science is self-correcting. Therefore, er, global warming is indeed true — it’s been through tons of peer review (as well as the blogs and media hype), and the science is no longer under debate! There are naysayers out there who believe that all scientists like to rub each other’s back and that some have the power to suppress the truth. Not so!

    As for Felisa’s microbes, there is just more work to be done to tell either way. They are intriguing, and no conclusion, yet. What NASA should do to apologize for all the hubbub is fund more studies, and give Felisa a job.

  51. Shaun Horton
    December 13, 2010

    Regardless of the reporting errors there is something that the media is truly missing. That is how inspiring the initial information was. I am a biology teacher and the initial announcement really brought me back into the current science news. My students will benefit from this. Discussions in classrooms all over the country occurred. Of coarse when Bill Nye speaks kids listen. He did speak and a young 12 year old mind was opened to new possibilities. Science became a priority to him that day.

    So now I can go back and teach my students about the scientific method again. But now I can add it peer review. Just another layer of information. This was actually good for the science in our classrooms.

  52. Carmen Drahl
    December 13, 2010

    Hi Ed- Thanks for mentioning my story, and thanks for a great post-mortem. This whole story points to how interdiciplinary research needs to be treated with extra attention- from scientists, who should build a circle of coworkers who can address gaps in a study from multiple angles- to journals, in assigning reviewers- to journalists, in finding the best possible mix of sources for comment. (I wish I’d called up the experts I talked to in my 2nd story for my first!)

  53. Paul from NH
    December 13, 2010

    The collapse of this story has been extremely disappointing to those of us who love the idea of alternate biochemistries… Damn it, I want my aliens! :-(

  54. Dagmar
    December 16, 2010

    @Sarah can’t figure out why we’re criticizing the authors “for publishing their findings in a high-impact journal, as is.”

    If a scientist does shoddy work, submits it to a journal (prestigious or not), and it gets published, first of all – does it matter? There’s no question that it does. The entire system of scientific publishing is designed to prevent this from happening.

    Now of course the system isn’t perfect so some shoddy work is going to get through. Who’s to blame? Both the authors and the journal bear responsibility. Authors should do what they can to ensure their conclusions follow from their results. That wasn’t done in this case. The peer review process should do the same and also fell down on the job. Getting through peer review does not absolve the authors from responsibility.

  55. Yikes
    December 17, 2010

    “If question remains about the voracity .. Voracity? Someone needs English 101.

  56. Ed Yong
    December 17, 2010

    It’s a crying shame that How Not to be a Wanker 101 isn’t an actual class. I guess it’s just assumed that people will pick it up as they go along…

  57. Kyle
    December 18, 2010

    I have to say that is a favorite blog of mine because of how accurately you report, contextualize, and hypothesize about interesting science. Keep up the great work!

  58. Telnet
    January 4, 2011

    Had I refereed the paper I would probably have asked for direct evidence for As in the DNA. I suspect this will not be forthcoming. Nevertheless, I am sure that this paper will do no harm to Science Magazine’s Impact Factor. It is too bad that there is no way to count negative citations…

Add Your Comments

All fields required.

Related Posts