Moderna And Merck (Revised!)

 BY DEREK LOWE

Update: Moderna contacted me about this post, and they have the following on-the-record statement about this trial and its statistics:

The primary efficacy endpoint and statistical plan of KEYNOTE-942 were pre-specified in the protocol. The trial was powered from the start to demonstrate 1-sided superiority of mRNA-4157/V940 plus KEYTRUDA versus KEYTRUDA monotherapy. Signal seeking trials are often powered to different thresholds than confirmatory Phase 3 trials to account for sample size. 

In KEYNOTE-942 the primary efficacy endpoint of 1-sided superiority was met at the first per protocol analysis (minimum of 40 events and 12 months follow-up for all patients). 

I'm glad to hear that this was the intention from the start - the worst sin in this area is to switch to one-sided statistics in order to make things look better, and that's not what happened here. I think that if this had been made clear in the press release, those of us in biopharma who raised eyebrows at the statistical treatment would not have been as alarmed as we were. The original text of yesterday's post is below (I've switched it to italics and added updates to it),because the general points about one-sided p tests are still valid. But I am now satisfied that Moderna (and their partners, Merck) are not trying to pull a fast one with the way this trial was run and reported. This makes me happier in several ways, but the biggest one is that this gives me more optimism that this therapy might demonstrate good Phase III results and lead to a new form of treatment for a lot of melanoma patients who are running out of options. I hope that mRNA-4157/V940 works, and while I'm at it, I hope that the similar immunotherapy ideas being developed its competitors work, too!

I should be happier about this press release than I am. You may well have heard that Moderna and Merck, who are partnering on an mRNA-based immuno-oncology therapy for melanoma, to be given along with Keytruda (pembrolizumab, their anti-PD1 antibody). The announcement says that their personalized cancer vaccine therapy, mRNA-4157/V940, met its primary efficacy endpoint and reduced the risk of recurrance or death by 44% in patients with advanced disease.

That’s a pretty spectacular number on the face of it, and as I say, I should be happy to see it. Initially, I was. But I then had the same reaction as a couple of experienced observers of such things, Matthew Herper at Stat and Frank David at Pharmagellan. That’s because when you read the fine print of this release, you see that the statistical significance is presented as a one-sided p value (something that Pharmagellan has already pointed out as a sign of trouble in such press releases). Why does that matter?

The p-values that you’re used to seeing are from two-sided tests (or they should be!) That’s a more robust way of handling the data, because a one-sided test is basically making the assumption that an effect in the opposite direction of what you want is either impossible, or would no bearing on the conclusions if it were to happen. And that just makes no sense for an investigational drug trial. Think of it this way: if you have two separate patients groups (treatment group and placebo, or treatment group and standard-of-care), then the null hypothesis is that there will be no real difference between the two - that is, the probability of success is identical. What’s the alternative hypothesis that you’re trying to see evidence for in the data? If it’s that the two groups will be different, that’s a two-sided test, because your treatment group could be different-better or different-worse. If you test only for the hypothesis that the treatment group will be better, that’s a one-sided test.

As that last link shows, you can get greater statistical power with a one-sided test per given number of patients (or alternatively reach the same statistical power by enrolling fewer patients than you’d need for a two-sided hypothesis). That power is only in that one direction, though! That’s only an advantage if you’re willing to ignore reality, and the reality is that (1) you don’t know that your therapy might not make people worse, because (2) there have been many trials, in many therapeutic areas and through many different mechanisms, where that exact thing has happened. This point is elaborated here, in a vigorous defense of two-sided statistics in drug research.

Here’s the thing: this issue is well known to designers of clinical trials. So when you see a one-sided p value quoted as evidence, that is not an oversight nor (let’s be real here) is it an honest mistake. I find it hard to believe that Merck and Moderna would have set up this trial from the start intending to use this measure, and switching to it at the end is not the way to behave. It is a deliberate choice taken to make the results look better. They quote a one-sided p value of 0.0266, and the only thing I can assume is that a two-sided p value would indeed be over the standard threshold for statistical significance. Which would not allow you to headline the whole thing with how the drug “met its primary efficacy endpoint” or to refer to it as a “statistically significant and clinically meaningful improvement”. (Update: see above - fortunately, it was set up that way from the start!)

Now who knows, it may turn out to be just that. But at this point, that is the literal truth of the matter: Who Knows? The companies are moving forward into a Phase III trial, which will surely have more patients and should provide a better statistical picture. I don’t think we’ll see those results announced as a one-sided p value. (Update: and it won't be - this was just for this particular Phase II, as Moderna clarified above). The FDA won’t stand for it, and neither should anyone else. I have ridiculed small biotechs for boasting about one-sided p values: what’s an appropriate response for companies like this? Moderna and Merck both should know better than to pull such a PR stunt. (Update: and I'm glad that it turns out that they do. . .!)

https://www.science.org/content/blog-post/moderna-and-merck-are-better-right