See a caveat about the very short trials at the end of this posting. They exaggerate exactly how short this trials could be, because this table is based on “normal approximation” to the actual probability distribution.
Yesterday, Dr. Anthony Fauci correctly stated the one and only way that vaccine clinical trials can end early. (Assuming that that the FDA was serious when it said that any vaccine approved for US use must pass all phases of its clinical trials.)
If the results are extremely good (or extremely bad), they could legitimately end the clinical trail before it was scheduled to end. Because, if that happens, they would legitimately and accurately be able to pass judgment on the vaccine, given the available data. And so, if the results are extremely good, they could approve the vaccine ahead of schedule.
That’s totally legit. And there is significant precedent for it. Clinical trials have been stopped before, and drugs given approval, if they are shown to be clearly effective against some life-threatening disease. In some sense, that’s every drug manufacturer’s dream. Google the phrase drug trial end early and you’ll see an entire scholarly literature on this topic.
I’m putting a marker down, in the form of this posting, for several reasons.
- It was refreshing to see a US public official get the math right. (Note that he said very good or very bad, which is the fully accurate statement.)
- Ditto, making a clear and unambiguous statement of fact.
- It will be interesting to see what actually happens, as election day approaches.
- I get the sense that we’re finally coming around to the Russian view of vaccine implementation (Post #773), only our bureaucracy can’t admit to that.
- I like math, and figured I’d use this as an opportunity to do some back-of-the-envelope statistical power calculations that I had been meaning to do anyway.
In other words, with point 5, I’d like to make some reasonable guess as to just how good the vaccine would have to be, to allow trials to end appreciably earlier than scheduled. Just so I have some sense of whether they are making things up or not, if (when?) they cut the Moderna (US) vaccine trial short later this year.
And I’ve now done that, shown at the top of this post. Fauci is absolutely right. If these vaccines are effective, it won’t take very long to demonstrate that. That’s what I find, calculating it from the ground up.
And so, if the vaccines are effective, and they cut the trials short, that’s not really a shortcut. That’s legit. And that’s not and excuse for not getting vaccinated.
And as an odd side note, the need for a statistical test effectively bars marginally-effective vaccines from being marketed. (Assuming they do their statistical tests legitimately.) If a vaccine is only 60% effective, you can eventually show that it’s “statistically significantly different” from the FDA 50% threshold. But with any luck the pandemic will be over by the time you do that.
Details follow.
Challenge trials are the joker in the deck
You may have missed this little news item that came out a couple of weeks ago. It said that the Federal government was brewing up batches of COVID-19 virus to use in “possible” human challenge trials.
Challenge trails: When you absolutely, positively need to know right now. If you’re really in a hurry to see if something works or not, you vaccinate, dose your subjects with the infectious agent, and see what happens. You “challenge” them with heavy exposure to the disease, typically one that would otherwise guarantee infection in an un-vaccinated individual.
With that approach, there’s none of this waiting around for nature to take its course. That’s more of a slam-bang, count the bodies and be done with it approach. At some significant risk to the participants. (Although, the writeup above suggests that it would still take months to set up and run, which I find hard to believe if it were done on an emergency basis.)
Challenge trials are not a new idea. They’re not even a new idea for testing COVID-19 vaccines. Even some of Our Statesmen in Congress figured this out, months ago.
But, interestingly, Fauci appears to dismiss that possibility completely. Meanwhile, “government scientists” (not otherwise identified in the news reporting) are creating the batches of COVID-19 that will only be needed if there are challenge trials. I wonder which one is right. And I wonder who the unnamed “government scientists” are.
But for now, I’m dismissing that possibility. If they were doing to do challenge trials, they should have done them months ago.
Approximate statistical power calculation
In Post #774, I showed some of basic arithmetic that explains why it is so hard to test a COVID-19 vaccine.
- It’s a deadly disease, so you can’t just take a few hundred people, give half a placebo, and expose them to COVID-19. (That’s a challenge trial, as discussed above.)
- Instead, you have to take tens of thousands of people, give half of them a placebo, and wait to see who gets infected in the normal course of business.
- Given the low rate of new infections, it takes considerable person-months of time to accumulate enough infections to provide usable data.
I did the arithmetic in that prior post. The US vaccine is aiming for a 30,000 person clinical trial. For the sake of argument, let’s assume that they instantly enrolled all 30,000 people. If they split that 50/50 (vaccine/placebo, good enough to make this point), at Virginia’s current infection rate (12 new infections per 100,000 population per day), you’d only accumulate about 50 infections per month in the placebo group.
But that might vary quite a bit, just by chance. Might be 20, might be 100. And if the vaccine works, you’d accumulate fewer than 50/month, on average, for the vaccine group. But that also would vary quite a bit, just by chance. And then, based on the small difference between those small counts, you’d have to decide how well you think the vaccine is working.
That is why the people conducting vaccine clinical trials:
- Chase down COVID-19 hotspots and recruit people there. That gives them more infections per person-month of exposure.
- Use huge samples (30,000 persons). That gives them more stable numbers (more “statistical power”) for a given infection rate.
The final thing you need to know is that the FDA set a floor of 50% effectiveness, which they defined as either preventing infection or reducing severity of infection in half of cases. A vaccine must be proven to do at least that well before they’ll approve it.
As an aside, that “or reducing severity of infection” may end up being weasel-wording of the highest order. I know how to count infected versus not. That’s black and white. But as far as I can tell, there’s literally no one legitimate way to calculate that second part about having reduced the severity of infection in X% of infected cases. Manufacturers will have to establish an arbitrary scale of severity (e.g., death = 1, hospitalized on vent = 2, hospitalized no vent = 3, symptoms but not hospitalized = 4, asymptomatic case = 5). And then perform some arithmetic on the counts of cases in each category, and from that conclude that they reduced the severity of infection in X% of cases. Near as I can figure, that last part is going to be some form of hand-waving. It will be some arbitrary method, applied to that arbitrary scale. That opens the opportunities for presenting your data in the most favorable light possible. To put it nicely.
In any event, the FDA 50% floor gives us the “target” for any statistical test. The clinical trial is going to come up with an estimate of effectiveness, and an estimate of how uncertain your are about that number, the so-called “95% confidence interval”. In everyday press, you’d see that expressed as a plus-or-minus range. (E.g., the estimate is 60% effective, plus or minus 3%. That would mean that, based on the data, you’re 95% sure it lies between 57% and 63%. And if you repeated that clinical trial many times, only one time in 20 would you see numbers like that, but the true (real) effectiveness of the vaccine was somewhere outside of that range.)
There’s also a little statistical cheat you can try to use here, called a “one tailed test”. It would be entirely inappropriate in this case (because a priori, you don’t know if you’re going to beat that 50% threshold or not), and I assume that legitimate scientists at FDA would not accept that. So I’m basing this on a standard two-tailed test.
With all that, and putting aside the part about reducing severity, it’s easy enough to mock up the resulting statistical test. To see just how much data you would need, in terms of person-days of exposure, to be able to exclude 50% effectiveness from the 95% confidence interval. Assuming you were able to perform you test in a place with, say 20 infections / 100K persons/ day.
So that’s my rough back-of-the-envelope word problem. Suppose you have a clinical trial with:
- 15,000 enrolled
- in an area with 20 infections/100K/day
- split 50/50 into vaccine and placebo groups. (The 50/50 split isn’t optimal, but this is just a rough calculation).
For a given true vaccine effectiveness of Y%, roughly how many days does that trial have to go on before you can exclude 50% effectiveness from the 95% confidence interval around your estimate of Y? In other words, how long until the vaccine passes the test?
Details that nobody cares about but me: I’m also going to use normal theory approximations here, because I know how to set up the problem in a spreadsheet that way. I don’t think that matters, as long as I have well over 30 infections in both placebo and vaccination groups. And I’m modeling the FDA threshold as a known number, when in fact, it’s going to be half the placebo group number, which is itself uncertain. I’m also ignoring the fact that the test is expressed as a ratio, which can complicate the statistics when both numerator and denominator are estimates (not known numbers). Finally, I’m ignoring that the trial doesn’t really start until the second month, because the US vaccine requires two doses, one month apart.
I built up this calculation using the standard formula for the variance of a binomial (yes/no) variable. Then I use the normal approximation and, in effect, calculate a set of standard t-tests. As mentioned above, that’s one of many things that’s not precisely correct about this calculation. But it should be close enough.
Results are given below. It’s no surprise that the better the vaccine, the less time it takes to prove it. (To exclude 50% effectiveness from the 95% confidence interval). What is surprising, to me, is how much overkill a 30,000 person trial is, if you expect the vaccine to be at all effective. I think this may explain why other vaccines, such as the British (Jenner Institute/Astrazeneca) vaccine are using more like 10,000 in their clinical trials.
As an extras for experts, some of those very short intervals would provide too few infections to allow legitimate use of the “normal approximation” that I used here. So those almost certainly overstate how short the trials would have to be, at the very bottom of the table. The rule of thumb is, you’d want to see at least 30 infections in at least one of the groups. At the placebo rate, then, that sets a floor of about three weeks to satisfy that additional constraint. But the gist of this is still correct. You only need a half-year of clinical trials if your vaccine doesn’t do much.
