Post #815: What if this is as good as it gets?

Source:  Immunogenicity and protective efficacy of influenza vaccination
Claude Hannouna, Francoise Megas,  James Piercy,  Virus Research 103 (2004) 133–138.

The importance of this graph will be clear about five paragraphs down.

At this point, with the Phase III trials of coronavirus vaccines well underway, even if they don’t have enough “statistical power” to do the formal statistical test, our public health bureaucracy ought to have a fairly good indication of how things are shaping up.

I’ve been waiting for any US public health leader to start leaking information on the likely effectiveness of the coronavirus vaccines.   Informally tossing some numbers out there, to get us prepped for the eventual formal announcement.

We just got our first indication today.  And, although the CDC Director broke the news gently, and indirectly, and with spin, if you paid attention, the news was clearly not good. Continue reading Post #815: What if this is as good as it gets?

Post #814: COVID-19 vaccines. The Chinese get it, the Russians get it. We don’t get it.

In Post #777 (and earlier), I talked about why the Russian strategy of early use of their vaccine made good sense.  And now, it turns out that the Chinese are doing the same thing as the Russians.  They just didn’t brag about it, the way the Russians did.  I guess I am unsurprised that the Chinese are doing the rational thing here.

In both cases, they are taking vaccines that have passed safety trials, but have not yet completed trials showing whether or not they work, and they are using them right now, for high-risk populations such as health care workers.

This is so rational, and so logical.  And yet, apparently so beyond the grasp of our government.  Either our elected leaders or our professional bureaucracy. Continue reading Post #814: COVID-19 vaccines. The Chinese get it, the Russians get it. We don’t get it.

Post #812: Vaccine distribution: The National Academy of Medicine plan


Source: Discussion Draft of the Preliminary Framework for Equitable Allocation of COVID-19 Vaccine, Committee on Equitable Allocation of Vaccine for the Novel Coronavirus, National Academies of Sciences, Engineering, and Medicine (September 1, 2020).

I almost missed this one, and thought I might bring it to your attention. Continue reading Post #812: Vaccine distribution: The National Academy of Medicine plan

Post #803: Hahaha. No, really? So make it 11/4/2020.

Cannot be rescinded. Not even questioned.

The whole COVID-19 vaccine thing just won’t quit.  We now have the officially-appointed spokesmodel saying that a November 1 deadline for having a vaccine in place is pure coincidence, and has nothing to do with election day.

If you’re just a civilian, you may not see this the way I see it.  But, if you’ve been in the game, you realize there’s a huge amount of slack in any deadline like that. 

They have no firm estimate of fill-in-the-blank here.  For example:  Of how effective the vaccine is.  Of how capable the manufacturers are of delivering large quantities by that date.  Of how capable state public health offices are of delivering it.  Of what temperature the vaccine must be kept at.  Of whether or not people are willing to get vaccinated with it.  Of who, exactly, will be eligible to receive the first doses.

And, plausibly, of which vaccine, exactly, they’re talking about.

And yet, November 1, 2020 is set in stone.  Cannot be rescinded, not even questioned.  When, to a fairly close approximation, you could avoid any question of the appearance of impropriety by postponing release to, say, a day or so after the election?

At some level, I just get tired of asking how dumb they think we are.  If you believe the party line on this, I feel sorry for you.  And I have a bridge in Brooklyn that I’d like to sell you.

Post #800: Vaccine by November 1. Who could possibly have guessed /s

Source:  Washington Post.

It’s not like the Administration hasn’t been signalling this for days now.  And, surprise, surprise, surprise, it’s scheduled for November 1.  Isn’t there some other important event that occurs in early November?  (Edit:  I forgot to mention that November 1 is a Sunday, almost as if nobody even bothered to look at the calendar before they announced the date.)

My best guess is that they’re going to offer this “emergency” authorized vaccine to health care workers only.  Nobody’s saying that, but that’s the only thing that makes good logical sense.  And that’s what the Russians did (e.g., Post #777).  The idea being that a) it’s known to be harmless, and b) even if it’s a dud (less than 50% effective, say, and so not meeting the standard that the FDA established) you can justify using it in a high-risk population if it reduces the odds of infection even a little bit.

And now some people will be offered the vaccine, and they have to come to some conclusion:  Is this real, or is it just political theater?

Let me walk though my analysis of it. Continue reading Post #800: Vaccine by November 1. Who could possibly have guessed /s

Post #798: Dr. Fauci and statistical power

See a caveat about the very short trials at the end of this posting.  They exaggerate exactly how short this trials could be, because this table is based on “normal approximation” to the actual probability distribution.

Yesterday, Dr. Anthony Fauci correctly stated the one and only way that vaccine clinical trials can end early.  (Assuming that that the FDA was serious when it said that any vaccine approved for US use must pass all phases of its clinical trials.)

If the results are extremely good (or extremely bad), they could legitimately end the clinical trail before it was scheduled to end.  Because, if that happens, they would legitimately and accurately be able to pass judgment on the vaccine, given the available data.  And so, if the results are extremely good, they could approve the vaccine ahead of schedule.

That’s totally legit.  And there is significant precedent for it.  Clinical trials have been stopped before, and drugs given approval, if they are shown to be clearly effective against some life-threatening disease.  In some sense, that’s every drug manufacturer’s dream.  Google the phrase drug trial end early and you’ll see an entire scholarly literature on this topic.

I’m putting a marker down, in the form of this posting, for several reasons.

  1. It was refreshing to see a US public official get the math right.  (Note that he said very good or very bad, which is the fully accurate statement.)
  2. Ditto, making a clear and unambiguous statement of fact.
  3. It will be interesting to see what actually happens, as election day approaches.
  4. I get the sense that we’re finally coming around to the Russian view of vaccine implementation (Post #773), only our bureaucracy can’t admit to that.
  5. I like math, and figured I’d use this as an opportunity to do some back-of-the-envelope statistical power calculations that I had been meaning to do anyway.

In other words, with point 5, I’d like to make some reasonable guess as to just how good the vaccine would have to be, to allow trials to end appreciably earlier than scheduled.  Just so I have some sense of whether they are making things up or not, if (when?) they cut the Moderna (US) vaccine trial short later this year.

And I’ve now done that, shown at the top of this post.  Fauci is absolutely right.  If these vaccines are effective, it won’t take very long to demonstrate that.  That’s what I find, calculating it from the ground up.

And so, if the vaccines are effective, and they cut the trials short, that’s not really a shortcut.  That’s legit.  And that’s not and excuse for not getting vaccinated.

And as an odd side note, the need for a statistical test effectively bars marginally-effective vaccines from being marketed.  (Assuming they do their statistical tests legitimately.)  If a vaccine is only 60% effective, you can eventually show that it’s “statistically significantly different” from the FDA 50% threshold.  But with any luck the pandemic will be over by the time you do that.

Details follow.

Challenge trials are the joker in the deck

You may have missed this little news item that came out a couple of weeks ago.  It said that the Federal government was brewing up batches of COVID-19 virus to use in “possible” human challenge trials.

Challenge trails:  When you absolutely, positively need to know right now.  If you’re really in a hurry to see if something works or not, you vaccinate, dose your subjects with the infectious agent, and see what happens.  You “challenge” them with heavy exposure to the disease, typically one that would otherwise guarantee infection in an un-vaccinated individual.

With that approach, there’s none of this waiting around for nature to take its course.  That’s more of a slam-bang, count the bodies and be done with it approach.  At some significant risk to the participants. (Although, the writeup above suggests that it would still take months to set up and run, which I find hard to believe if it were done on an emergency basis.)

Challenge trials are not a new idea.  They’re not even a new idea for testing COVID-19 vaccines.  Even some of Our Statesmen in Congress figured this out, months ago.

But, interestingly, Fauci appears to dismiss that possibility completely.  Meanwhile, “government scientists” (not otherwise identified in the news reporting) are creating the batches of COVID-19 that will only be needed if there are challenge trials.  I wonder which one is right.  And I wonder who the unnamed “government scientists” are.

But for now, I’m dismissing that possibility.  If they were doing to do challenge trials, they should have done them months ago.

Approximate statistical power calculation

In Post #774, I showed some of basic arithmetic that explains why it is so hard to test a COVID-19 vaccine.

  • It’s a deadly disease, so you can’t just take a few hundred people, give half a placebo, and expose them to COVID-19.  (That’s a challenge trial, as discussed above.)
  • Instead, you have to take tens of thousands of people, give half of them a placebo, and wait to see who gets infected in the normal course of business.
  • Given the low rate of new infections, it takes considerable person-months of time to accumulate enough infections to provide usable data.

I did the arithmetic in that prior post.  The US vaccine is aiming for a 30,000 person clinical trial.   For the sake of argument, let’s assume that they instantly enrolled all 30,000 people.  If they split that 50/50 (vaccine/placebo, good enough to make this point), at Virginia’s current infection rate (12 new infections per 100,000 population per day), you’d only accumulate about 50 infections per month in the placebo group. 

But that might vary quite a bit, just by chance.  Might be 20, might be 100.  And if the vaccine works, you’d accumulate fewer than 50/month, on average, for the vaccine group.  But that also would vary quite a bit, just by chance.  And then, based on the small difference between those small counts, you’d have to decide how well you think the vaccine is working.

That is why the people conducting vaccine clinical trials:

  • Chase down COVID-19 hotspots and recruit people there.  That gives them more infections per person-month of exposure.
  • Use huge samples (30,000 persons).  That gives them more stable numbers (more “statistical power”) for a given infection rate.

The final thing you need to know is that the FDA set a floor of 50% effectiveness, which they defined as either preventing infection or reducing severity of infection in half of cases.  A vaccine must be proven to do at least that well before they’ll approve it.

As an aside, that “or reducing severity of infection” may end up being weasel-wording of the highest order.  I know how to count infected versus not.  That’s black and white.  But as far as I can tell,  there’s literally no one legitimate way to calculate that second part about having reduced the severity of infection in X% of infected cases.   Manufacturers will have to establish an arbitrary scale of severity (e.g., death = 1, hospitalized on vent = 2, hospitalized no vent = 3, symptoms but not hospitalized = 4, asymptomatic case = 5).  And then perform some arithmetic on the counts of cases in each category, and from that conclude that they reduced the severity of infection in X% of cases.  Near as I can figure, that last part is going to be some form of hand-waving.  It will be some arbitrary method, applied to that arbitrary scale.  That opens the opportunities for presenting your data in the most favorable light possible.  To put it nicely.

In any event, the FDA 50% floor gives us the “target” for any statistical test.  The clinical trial is going to come up with an estimate of effectiveness, and an estimate of how uncertain your are about that number, the so-called “95% confidence interval”.  In everyday press, you’d see that expressed as a plus-or-minus range.  (E.g., the estimate is 60% effective, plus or minus 3%.  That would mean that, based on the data, you’re 95% sure it lies between 57% and 63%.  And if you repeated that clinical trial many times, only one time in 20 would you see numbers like that, but the true (real) effectiveness of the vaccine was somewhere outside of that range.)

There’s also a little statistical cheat you can try to use here, called a “one tailed test”.  It would be entirely inappropriate in this case (because a priori, you don’t know if you’re going to beat that 50% threshold or not), and I assume that legitimate scientists at FDA would not accept that.  So I’m basing this on a standard two-tailed test.

With all that, and putting aside the part about reducing severity, it’s easy enough to mock up the resulting statistical test.  To see just how much data you would need, in terms of person-days of exposure, to be able to exclude 50% effectiveness from the 95% confidence interval.  Assuming you were able to perform you test in a place with, say 20 infections / 100K persons/ day.

So that’s my rough back-of-the-envelope word problem.  Suppose you have a clinical trial with:

  • 15,000 enrolled
  • in an area with 20 infections/100K/day
  • split 50/50 into vaccine and placebo groups.  (The 50/50 split isn’t optimal, but this is just a rough calculation).

For a given true vaccine effectiveness of Y%, roughly how many days does that trial have to go on before you can exclude 50% effectiveness from the 95% confidence interval around your estimate of Y?  In other words, how long until the vaccine passes the test?

Details that nobody cares about but me:  I’m also going to use normal theory approximations here, because I know how to set up the problem in a spreadsheet that way.  I don’t think that matters, as long as I have well over 30 infections in both placebo and vaccination groups. And I’m modeling the FDA threshold as a known number, when in fact, it’s going to be half the placebo group number, which is itself uncertain.   I’m also ignoring the fact that the test is expressed as a ratio, which can complicate the statistics when both numerator and denominator are estimates (not known numbers). Finally, I’m ignoring that the trial doesn’t really start until the second month, because the US vaccine requires two doses, one month apart.

I built up this calculation using the standard formula for the variance of a binomial (yes/no) variable.  Then I use the normal approximation and, in effect, calculate a set of standard t-tests.  As mentioned above, that’s one of many things that’s not precisely correct about this calculation.  But it should be close enough.

Results are given below.  It’s no surprise that the better the vaccine, the less time it takes to prove it.  (To exclude 50% effectiveness from the 95% confidence interval).  What is surprising, to me, is how much overkill a 30,000 person trial is, if you expect the vaccine to be at all effective.  I think this may explain why other vaccines, such as the British (Jenner Institute/Astrazeneca) vaccine are using more like 10,000 in their clinical trials.

As an extras for experts, some of those very short intervals would provide too few infections to allow legitimate use of the “normal approximation” that I used here.  So those almost certainly overstate how short the trials would have to be, at the very bottom of the table.  The rule of thumb is, you’d want to see at least 30 infections in at least one of the groups.  At the placebo rate, then, that sets a floor of about three weeks to satisfy that additional constraint.  But the gist of this is still correct.  You only need a half-year of clinical trials if your vaccine doesn’t do much.

Post #791: Winging it on vaccines?

It’s hard to tell whether the most recent statement on COVID-19 vaccines, out of the FDA, represents a change of policy or not.  On the face of it, it seems to.

Three weeks ago, the FDA said that every vaccine must pass through all phases of testing before being approved (Post #773).  That certainly seemed clear and unambiguous, with no possibility of exception.

Yesterday, it looks like FDA did an about-face, and said they’d approve a vaccine before testing was completed.  Details are given below.

I wonder which one is correct, if either?
Continue reading Post #791: Winging it on vaccines?

Post #774: How about some vaccine acid tests?


Image source:  Source:  Grainger Industrial

What I’m suggesting is that proposed vaccines that have passed Phase I (safety) testing ought to be tested, right now, in places where large oubreaks are common.  While those outbreaks are happening. Places like prisons, meat-packing plants, nursing homes, hospitals, and other high-risk environments.

If nothing else, if these vaccines have some effectiveness, you’d be doing the workers and inmates in those situations a favor.  But in addition, there’s a good, solid arithmetic behind doing such acid tests:  You’d know your answer to “does this vaccine work” far faster than you would using standard Phase II drug testing, which relies on community members to volunteer, and then be randomized into treatment and control groups.  And that would be doing us all a favor.

The only hard part is how to do it in a way that actually generates a usable test of the vaccine.  And even that part isn’t rocket science.  It just has to skirt an ethics question and get some cooperation among state, federal, and private-sector entities.

Continue reading Post #774: How about some vaccine acid tests?

Post #773: COVID-19 vaccine: 1: The Russians have it right …

2:  The US is screwing it up.

3:  The major drug companies might be able to save us, despite ourselves.

4:  But the stupidity of The American People may yet snatch defeat from the jaws of victory.

This is one of a series of posts about vaccines for viral illness, and about a COVID-19 vaccine.  See Post #741 for a quick run-through of flu vaccine effectiveness.  My posts tend toward TL/DR, so in this post I’m only going to address Item 1:  The Russians have it right.

To cut to the chase, this is basic game theory.  Suppose you had a COVID-19 vaccine candidate that might be effective, but might be just a harmless placebo.  That is, it has passed Phase I (safety) clinical trials.  What’s the smartest thing you could do with it?  Continue reading Post #773: COVID-19 vaccine: 1: The Russians have it right …