Yet another Science paper retracted today. Nothing much new unfortunately (sting!). Except that for me and my colleagues this was not just another example of non-reproducible work. We planned our research projects around it. So here are my thoughts about this: just in time for Halloween.
There has been much talk about the problem of reproducibility of data and the rise in retractions. The discourse is mostly centered around the perpetrators and the negative impact this sloppy or knowingly flawed science has on the industry and the perception of scientific endeavor in society. Who did this awful study? What reviewer did not catch this missing control and the missing/sloppy stats? Why did they do it? Why don’t they admit their mistake? What’s wrong with peer-review? Well….much is wrong with the industry and many people have great ideas how to tackle some of these issues.
What is usually missed from the discussion is the impact such dubious science can have on young, early career scientists if they are at the receiving end of non-reproducibility. What do you do if you come to a famous (or not so famous) lab, you get a project, which is based on fantastic data and high impact publications, BUT you cannot reproduce it? How do you approach this issue? What does it mean for you and your career if your project just goes to shreds because it’s simply based on bad data? Do you get another chance? Will you be forever associated with this flawed data and your reputation damaged? Are you going to be so disillusioned with science to the point that you want to leave academia? Do you try to fix the problem or silently move on? Do you try to make it work? Do you tell the boss and explain your situation? Or is it you? Can you just not get it to work? Can it really be that this is all wrong? Why cannot I reproduce this data? Can this BE???
I think everyone can appreciate the complexity of the issue. Many people I talked to had their own experience to share. Here is what’s frightening: sloppy science and misconduct, I thought, is something you read about in journals and not something I would experience myself. This would never affect me or close friends in other labs, who are all great scientists in my eyes. I was wrong.
We young, early career scientists in our lab went through something very serious and exemplary recently. In mid 2011 to beginning 2012 we just joined the lab of Pamela Ronald during a huge turnover of people. Everyone of the “old crew” of many years, sometimes more than 8, left or were about to leave around the time we came in. We were all super excited to get going and to push the borders and boundaries of science /// Thanks Saul for this awesome expression./// However, no matter how hard we tried, our own data did not add up and we could not make much sense of it in the view of the lab’s recent immunity-related publications, which appeared in Science and Plos One. /// Here is a three sentence summary: The lab cloned an amazing immune receptor XA21 in rice that confers resistance to the most important pathogenic bacteria on rice (Xanthomonas oryzae pv. oryzae) in 1995. In 2009 they published the identification of the bacterial protein Ax21 as ligand of this immune receptor. In 2011 they reported the function of this bacterial protein Ax21 as a novel cell-cell communication factor (also referred to as quorum sensing factor) in the bacteria./// We, a group of concerned early career scientists in the lab, brought this discrepancy regarding the 2009 and 2011 studies to the attention of the PI and explained to her what we thought was wrong. A long, intense, careful, multifaceted, repetitive and diligent truth-searching endeavor started. I won’t bore you with the details. In short, we were not able to reproduce the core findings of both Ax21-related papers. Of course, this was very distressful for everyone involved especially as other people published similar work in relation to the bacterial protein Ax21. /// I am not sure what this says about the current state of science. I find it very concerning./// Kudos to Pamela Ronald, who reached the only scientifically correct decision and retracted both papers. She obtained much praise for this brave step. For sure, not every PI would have followed through in a similar situation. Pamela Ronald’s decision became the topic of the inaugural post in the new category “Doing the right thing” on the famous “retraction watch” blog. You can also read her perspective on how this whole story unfolded in her guest post on the Scientific American blog. I can only agree and applaud Pamela Ronald on making the right decision and retracting these two papers. For sure, it was not easy.
What I would like to highlight is the importance to clear the scientific record once mistakes in publications are discovered. Mistakes will always happen yet they need to be corrected. People should not fear to clear their scientific record but rather be rewarded for doing so. I am sure every reader of this blog knows of multiple scientific skeletons in people’s closets. It takes much courage from the PI’s perspective to bring these mistakes out in the open. They potentially have much to lose….BUT the PI will have the time to correct the situation. On the other side what’s going to happen with early career scientists, who got into one of these situations without their own wrong doing? Who is routing for them? What happens if the PI does not retract the paper(s) and simply disregards the concerns? I know of least of one situation in which a good friend of mine had exactly this happening to her. Finally she simply left academia, because there was no support for her and no way out. Young, early career scientists can be treated like a dispensable commodity if they don’t play along. Of course, this is the worst-case scenario, yet once you are trapped in the non-reproducibility hole you will lose a lot of time no matter what. You might be asked to repeat experiments multiple times, generate the same material anew, and, in general, spend a lot of time on research that is hardly publishable ///Ha!! no impact factor for retractions///. No one can give you this time back in your career and many people might not understand why you are not moving forward. Well, you simply cannot build skyscrapers on swampland. Only once everything is cleared up will you be able to move on with your career without losing too much. This is the only way out for the early career scientists, and PIs should be aware that they are not only dealing with their own careers. ///In this regard again thanks to Pamela Ronald for taking the courage.///
It is not only the PI’s but also the early career scientists’ significant efforts and courage that leads to the clean-up of the scientific record. Be diligent in these situations. Talk critically with your co-workers in the lab. Verify your own stock and everything given to you. Perform the right controls. Ask for advise from other senior colleagues. Follow your understanding of science as a truth-finding endeavor. This journey won’t always be linear and sometimes mistakes happen. Deconstruct them, correct them, learn from them and build something new. In my opinion, this is what truly makes a reputable scientist.
P.S.: So let’s figure out what’s really going on with this amazing rice immune receptor XA21. /// For the “love of wisdom”///.
lushgreengrassatafridayafternoon 
Serious scientists spend a lot of time working, not just at the frontiers of science but at the frontiers of their abilities. We’re certain to make mistakes, sometimes big ones, and we need a scientific process that will minimize the harm.
Distant echoes of my own life, where my first postdoc was based on a flawed FEBS Letts and thesis. What saved me was stubbornness, being young with no ties, a desire to explore the world and follow my scientific curiosity. Having sorted out the mess in 3 years and being rather naïve, I asked for a reference from the PI. I apparently got a multipage letter extolling my virtues and productivity (my CV was pathetic) and I got the next postdoc! I used the old and new PIs as references three years later and got the job I still have.
So quite a lot depends on references – at least in the UK – and on interview. With a PI more than deserving the title, you should have no problems with your reference.
A lot also depends on the hiring committee. If they are hiring for instant #Glampub, then you may not get the job. If they are hiring with a 30 year vision (roughly your career) you may well get the job. Why? Because this recent experience illustrates a key quality necessary for scientists. In essence, you have been tested by fire and have no burns. Where do you want to work? I would suggest that the second type of institution is by far the most interesting and fun place. You will probably work harder (more time with students, researchers and other PIs, less time flaunting your self-importance), but when you look back across a few decades you will have a much greater sense of satisfaction.
Pingback: I’ve got your missing links right here (12 October 2013) – Phenomena: Not Exactly Rocket Science
Thank you for the sincere, well-thought article above. As someone who has observed repeated statistical massaging & omission of data- meaning anything that doesn’t support the hypothesis gets called an “outlier” & dropped from the numerical analysis, or even worse, what I call the “black hole effect,” which is also a type of omission, though much more egregious, where entire experiments, one after another, are not included in a lab’s published body of work because none of their data-points support the lab’s hypothesis du jour. I’ve heard six months of experiments shrugged off with a “yah, those didn’t work out…”met with nary a peep from an unscrupulous PI. Based on witnessing these phenomena at multiple types of labs (medical, organismal, agricultural), combined with the voodoo which passes for troubleshooting molecular biology experiments, which have incredible problems with reproducibility due to finicky reagents & a global lack of standardization with complicated assays, I sincerely question the validity of anything which hasn’t been reproduced in thousands of replicates in a field system, or an animal model, or some other definitive assay which doesn’t ask me to look at a Western or Northern blog in isolation as definitive proof. The physiologists have it right- molecular biology in isolation hasn’t really got the greatest track record outside of clarifying taxonomy & identifying putative regions of gene/ gene function. The track record of classically trained physiologists in all life science fields over a 50-year span vastly outstrips that of the ~three decades of mainstream molecular biology. Back to the data problems, PIs huddled in their offices writing grants, never taking the time to go into their lab & thumb through notebooks (which rarely exist anymore in some places), have led to the trainees driving the bus, or, in other words, the blind leading the boss.
Pamela Ronald is an inspiration. I would wear a modestly priced T-shirt with her mug & name on it. People like her, along with her students, may well be the salvation of science.
This is a great blog and I applaud you and your fellow early career scientists for bringing this surely complicated and messy evaluation of previous data to the eyes of your PI as well as to believe in yourselves and push each other on to do the right thing and stand for science. I have been working for years in the field of crop disease resistance and am very familiar with the work of the Ronald lab, and RLKs in general. I know that several conclusions rely on assays that are difficult and require years of optimization. What you and your colleagues have achieved is truly commendable and deserves the utmost praise and respect. You are exactly the kind of postdoctorals and phd students we are all looking for. The PI will only be able to make a decision based on their team, this is how modern day science works. PIs don’t re-evaluate experiments physically in the greenhouse or laboratory benches anymore, so they need to rely on courageous lab members. I want to praise another aspect that others commenting may have missed. For you and your fellow postdoctorals to believe in yourself after not reproducing data is the first step towards becoming a critically thinking and reliable scientist. That is the most difficult thing and shows the true mettle of a great scientist. I think that this blog should be featured on platforms that use your example as role models for other junior scientists. Pamela Ronald should be so proud of you all for being in her team and being the saviours of a tricky situation. Congratulations. True merit is always rewarded, even if it takes time.
I was impressed by Ronalds sincere and insightful “anatomy of a retraction” on the Scientific American blog , and it’s very interesting to read your postdocs’s perspective on the matter.
I have a question though, for the sake of understanding:
You write “We …brought this discrepancy regarding the 2009 and 2011 studies to the attention of the PI ”
So, do you say the first clue that something must be wrong came through carefully reading and comparing the two published papers? In Ronald’s version I got the impression, the whole thing started to crumble first when the lab tried to reproduce the results.
If already the published papers would indicate dicrepancies, one would indeed wonder a bit why none else – the reviewers of the 2011 paper for example – had pointed it out at the time of publication.
(But I may misunderstand what you said , therefore the question).
Thanks
Thanks Hans for your question I will try to clarify the point.
I think it is very difficult to spot mistakes of the mislabeling-type in the review process, because the reviewer assumes that all presented genetic stocks and chemicals are correct. However the 2011 paper was initially under extensive review in Science, but finally got rejected. Pamela Ronald wrote a blog post about the whole story once the 2011 paper was accepted in PlosOne here –> http://phylogenomics.blogspot.com/2012/01/cracking-microbial-code-pam-ronald.html. At least one commentator pointed out that the role of Ax21 as quorum sensing factor was highly unlikely as it was predicted to form a beta-barrel membrane porin (check out the comments). In addition, as Pamela Ronald pointed out at the Keystone conference, one of the clever young post-docs figured out, by simply looking at the primary amino acid sequence, that Ax21 has a Sec-secretion signal instead of a ‘type-I’. He also went on to show that Ax21 is processed in E.coli that lacks the type-I system. Of course much more experiments were done that let to the retractions.
Thanks for your response, that’s interesting.
It’s almost cruel now to read the comments of the structural guy below the post you mentioned. In hindsight, a sectreted porin sounds quite extraordinary, maybe demanding extraordinary evidence.
But that’s just the hindsight-perspecticve, of course. It’s quite common to have some unsettled questions at the time of publication, after all.
Pingback: Report on a retraction: Minding the scientific skeleton in the closetWeeding the Gems
Pingback: Behind the Scences: The people who debunked Ax21 | lushgreengrassatafridayafternoon
Pingback: Discovery and Redemption emerge from a scientific mistake | lushgreengrassatafridayafternoon