Over the past 12 years, Ben Mol, MD, PhD, has turned a passing interest in research integrity into a nearly full-time job. The ob/gyn at Monash University, in Melbourne, Australia, estimates that he spends roughly 30 hours per week sifting through published articles ― particularly those reporting randomized, controlled trials ― not for breakthroughs in his field but for signs of manipulated data, to challenge these articles with editors and to raise awareness in his field. He finds problems with dismaying frequency. The fruits of his efforts appear regularly on Retraction Watch, a blog devoted to covering retractions from the scientific literature, and he has written several journal articles about his findings. (He estimates that he is responsible for at least 90 retracted articles and 60 expressions of concern so far.) Mol also was the subject of a recent article in The Economist about fraud in medicine.
Dr Ben Mol
Medscape Medical News spoke recently with Mol about his work as a data sleuth and the state of medical publishing.
Medscape: How did you get interested in the area of "research into research"? Has this been a problem specific to reproductive medicine and ob/gyn?
Mol: My core business for nearly 30 years now has been making medical knowledge by doing randomized clinical trials [RCTs]. And I'm quite passionate about it. I was a bit naive when I was young. Then I thought that we would do a trial and translate immediately to practice. I realize now that there are a lot of hurdles, but I still think that an informed world is better than an uninformed world. If people know the evidence or if the evidence is considered when they make clinical decisions, I think we have a better world.
With that in mind, I started to find out about 10 years ago, for the first time, that there were studies that were almost not true, so to say. Until that moment, we had only considered strong and weak studies, and then you had the GRADE system, trying to weigh the strength of the evidence. But few people were thinking that the data underlying what we were reading would not be true. And then when we focused on it, we saw it on a much larger scale.
If you see medical knowledge as a clear river that we provide people to drink clean water from, then if you see that someone else is just polluting the water with all kinds of dirt, that is a problem. It affects your whole system's intuition.
Medscape: Although it wasn't your intent starting out, in your work as a "data sleuth," you've found more reason for concern in some areas of the world than others: Egypt, Iran, India, and China. What should readers of the ob/gyn literature know about these geographic hotspots? Should they be particularly wary of all published studies coming from these countries?
Mol: Racial profiling is a very sensitive thing in this day and age. I think a lot about it, and I discuss it with other people. But the total number of randomized clinical trials [in ob/gyn and reproductive medicine] only comes from a limited number of countries. The reason I'm not looking at Argentina or Chile or the Philippines or Mongolia is they don't have that large volume of RCT teams. And unfortunately, within those 15 main producers of RCTs, there are four or five countries that produce a lot of our citations. And they don't have good governance in place for their major trials.
One of the reasons I can detect this is that I'm an obstetrician/gynecologist and I'm a researcher, which gives me, I think, an advantage over other people in detecting anomalies or strange things ― things like birth weights that are strange or gestational ages that are strange. But I don't see those anomalies in neurology or psychiatry, so it's difficult for me to assess this in other fields. And there's also a time issue. But if I look at the work of John Carlisle in anesthesiology, for example, who is completely independent from what I do, he sees exactly the same phenomenon.
Medscape: You've estimated that you spend 30 hours per week investigating suspect articles. That's a huge amount of time. Is it sustainable?
Mol: Not in the long run. But we are definitely not alone, and there's definitely a different mindset in our field. You see an enormous drop in output from particular countries as compared to 4 or 5 years ago. While we can't prove causation, I take a little bit of satisfaction as people are more aware of the problem that we flagged.
Prevention is obviously the best thing. There are other people who have other initiatives in prevention. There's going to be a paper in the British Journal of Obstetricians and Gynecology soon about research integrity. Other people are considering this item, and we are seeing all kinds of initiatives in awareness that can change this problem.
Medscape: You've written a new article with John Ioannidis, MD, DSc, for Fertility and Sterility about increasing the trustworthiness of the medical literature in which you seem to be arguing that hype is a bigger problem in science than fraud. (Ioannidis' 2005 article in PLOS Medicine, "Why Most Published Research Findings Are False," is among the most influential scientific articles of the 21st century.)
Mol: I think there are two problems here. There is a blunt fabrication, and then there is the promotion of your work in a positive way but the underlying data are true. But there is a circle of publishing and being well cited, getting funding, and being able to publish again. And that circle, I think, is where most of us try to make their work look a little bit more positive than it really is.
Medscape: There seem to be two sides to this problem. One is the supply side ― the responsibilities of not only the researchers but also the publishing apparatus to police the literature. And then there's the demand side and the expectations and proper skepticism readers must bring to the process of consuming the literature. Which side, in your view, bears the heavier burden?
Mol: I think that the big ball at present sits with editors and publishers. If you would have asked me 2 years ago, I would have said: Well, if we bring these problems into the spotlight, the journals will start cleaning this mess up. But they don't. They delay. They don't investigate and their processes are completely not transparent.
Medscape: Your work seems to indicate that the problems, at least in reproductive medicine, mainly concentrate in the lower-tier journals. Is that a fair assessment?
Mol: Yes, I would say that in the lower-impact journals, you're really surprised at what they publish. And you have to ask yourself if people have really looked at the papers. Some of the research that is published is just nonsense. In some studies, you see impossible timelines, so babies are born after the paper was submitted, or the paper was submitted before the study could possibly have been completed. This kind of thing generally is not happening at higher-impact journals. But the problem remains because these bad papers are included in meta-analyses, and then if Cochrane [the Cochrane Library] picks them up, nobody sees whether a thousand patients came from the high-quality paper or from the problematic one.
Adam Marcus is editorial director, primary care, at Medscape and the co-founder of Retraction Watch.
No comments:
Post a Comment
Note: Only a member of this blog may post a comment.