The Problem with COVID-19 Clinical Trials

By Derek Lowe

Let’s talk about a painful subject. I am of the opinion – and I’m far from alone – that the most reliable way to determine if a possible therapy has any usefulness is a randomized, double-blinded controlled clinical trial. I can be a bit more specific than that, even: let’s make that “a trial that is run with sufficient statistical power to have a good chance of providing a meaningful readout”.

The worldwide coronavirus pandemic has featured some well-run trials that have truly advanced our knowledge of the disease and how to treat it. But it has featured far, far more garbage. That word was chosen deliberately. There have been too many observational trials, too many uncontrolled (or poorly controlled) ones, too many open-label ones, and above all, there have been way too many trials whose number of patients would be insufficient to tell us much of anything even if everything else had been run properly.

I am not revealing any hidden tricks of the trade here. Clinical trial design is a subject with a very large literature, and there are any number of people and organizations who can provide useful guidance on both its theoretical and practical aspects. Among these aspects are the calculations that should be made for how many patients a trial is likely to need to be well-powered enough for a clean read on its clinical endpoints. You can start to learn the basic outlines of the subject online. Now, that’s not to say that it’s an easy subject to get ahold of. You’re going to have to estimate some of your key parameters as well as you can, among them what you think the effect size of your treatment might be, what the patient-to-patient variability might be like, the time course of treatment that might be needed, and more. Just picking the proper clinical endpoints is a subject all in itself (and it’s one that can have a huge effect on a trial’s design and on its chances for success). And at the other end of things, your inclusion criteria and patient enrollment process is a place for serious thought, too. Who should be evaluated (or definitely not evaluated) in your trial, and how long will it take you to round those people up? Where are you thinking about doing all this, anyway?

There are a wide variety of trial designs out there as well, and you can find yourself sorting through some that are clearly inappropriate to the problem at hand, some that would be great if you had about ten times as much money and time as you do, and several that at first glance look like they could all work out, but which have real-world differences that it’s crucial that you be aware of. You would be well advised to consult with experience practitioners before you start, to make sure you’re on the right track.

Unfortunately, underpowered, badly-run, and badly designed trials have been with us for a long time. Here are some well-justified concerns from 2002, for starters, and various fields of clinical research undergo periodic bouts of soul-searching over the years about these issues. But the pandemic year has really made some of our problems more obvious. Not only do we have trouble with badly run trials, but mixing in with that is a bandwagon effect. Clinicians all over the world just piled onto some of the coronavirus ideas, and kept piling on for months and months and months. Think, for example, about the hydroxychloroquine situation. Now, I still get messages condemning me as an implacable, irrational foe of the One True Coronavirus Therapy. But it’s worth remembering that I started out as a “Huh, I don’t know how that would work, but let’s look into it” person, which I really think should be the default setting. And in that spirit, I was all for running trials and getting more hard data.

But what did we get? A search through clinicaltrials.gov for “hydroxychloroquine|coronavirus” gives you 113 trials. What’s more, thirty-six of those are still listed as “recruiting patients”. This is ridiculous, but it’s not amusing. There are some large, well-controlled data sets available that indicate that HCQ is very likely not a useful therapy, but as you can see, there are also dozens of other smaller ones that say Yes! No! Maybe! Sorta! Kinda! Kinda Not! Depends! Could Be! Who Knows? And that adds up not just to a lack of knowledge, it turns into an actual hindrance to knowledge as you try to sort through the data. The heap of fuzzy indeterminate results also fuels the extrascientific political and cultural arguments about the drug, since everyone can find some sort of “support” for whatever opinion they might have.

You have to think that there were other therapies that deserved a look in the clinic as compared to the forty-third, sixty-seventh, or ninety-eighth hydroxychloroquine study. You’ll recall that for a while, HCQ ended up mixed into other clinical trials just because everyone wanted it or imagined that it was some sort of standard of care, and that did no one much good, either. Now, HCQ isn’t the only offender, but it’s a big one, and I think it illustrates what we should try not to do next time.

How, then, should we try not to do that? It’s not like the US (to pick a big example) has a National Clinical Trial Authority that passes judgment on these things. To be honest, the downsides of having such an agency might worry me even more. But letting everyone go into Headless Poultry Mode and pile up overlapping crap in the clinic isn’t such a good way to go, either. You would hope for a little more coordination among major medical research centers, and you’d also hope for some local university/research hospital review boards to be aware that greenlighting the East Porkford Covid-19 Treatment Study with 47 patients isn’t really going to advance medical science very much. Especially when it’s covering the same ground as the trials kicking off in Mashed Potato Falls, Rancho Malario, and Kidneystone Pass. But I’m being unfair to East Porkford – some of these lackluster trials were conducted at larger institutions that should have known better. The way we’re set up, it’s down to the review boards and the sources of funding to police things better, and to keep their heads while all about them are losing theirs.

And it’s also down to the NIH and the CDC to lead the way more than they did during 2020. The RECOVERY trial in the UK has been an example of what can be accomplished in that line. The NIH has helped run some good trials, but we’ve had nothing that comprehensive in the US as compared to the UK effort, and I really wish we had. I fear that some day, eventually, we’re going to have a chance to do better, and I hope that we take it.

https://blogs.sciencemag.org/pipeline/archives/2021/03/17/the-problem-with-covid-19-clinical-trials