Category: Analysis

Anything that I have tested empirically, or explain with detailed calculations, or just generally bother to get all the facts straight.

Post #798: Dr. Fauci and statistical power

See a caveat about the very short trials at the end of this posting.  They exaggerate exactly how short this trials could be, because this table is based on “normal approximation” to the actual probability distribution.

Yesterday, Dr. Anthony Fauci correctly stated the one and only way that vaccine clinical trials can end early.  (Assuming that that the FDA was serious when it said that any vaccine approved for US use must pass all phases of its clinical trials.)

If the results are extremely good (or extremely bad), they could legitimately end the clinical trail before it was scheduled to end.  Because, if that happens, they would legitimately and accurately be able to pass judgment on the vaccine, given the available data.  And so, if the results are extremely good, they could approve the vaccine ahead of schedule.

That’s totally legit.  And there is significant precedent for it.  Clinical trials have been stopped before, and drugs given approval, if they are shown to be clearly effective against some life-threatening disease.  In some sense, that’s every drug manufacturer’s dream.  Google the phrase drug trial end early and you’ll see an entire scholarly literature on this topic.

I’m putting a marker down, in the form of this posting, for several reasons.

  1. It was refreshing to see a US public official get the math right.  (Note that he said very good or very bad, which is the fully accurate statement.)
  2. Ditto, making a clear and unambiguous statement of fact.
  3. It will be interesting to see what actually happens, as election day approaches.
  4. I get the sense that we’re finally coming around to the Russian view of vaccine implementation (Post #773), only our bureaucracy can’t admit to that.
  5. I like math, and figured I’d use this as an opportunity to do some back-of-the-envelope statistical power calculations that I had been meaning to do anyway.

In other words, with point 5, I’d like to make some reasonable guess as to just how good the vaccine would have to be, to allow trials to end appreciably earlier than scheduled.  Just so I have some sense of whether they are making things up or not, if (when?) they cut the Moderna (US) vaccine trial short later this year.

And I’ve now done that, shown at the top of this post.  Fauci is absolutely right.  If these vaccines are effective, it won’t take very long to demonstrate that.  That’s what I find, calculating it from the ground up.

And so, if the vaccines are effective, and they cut the trials short, that’s not really a shortcut.  That’s legit.  And that’s not and excuse for not getting vaccinated.

And as an odd side note, the need for a statistical test effectively bars marginally-effective vaccines from being marketed.  (Assuming they do their statistical tests legitimately.)  If a vaccine is only 60% effective, you can eventually show that it’s “statistically significantly different” from the FDA 50% threshold.  But with any luck the pandemic will be over by the time you do that.

Details follow.

Challenge trials are the joker in the deck

You may have missed this little news item that came out a couple of weeks ago.  It said that the Federal government was brewing up batches of COVID-19 virus to use in “possible” human challenge trials.

Challenge trails:  When you absolutely, positively need to know right now.  If you’re really in a hurry to see if something works or not, you vaccinate, dose your subjects with the infectious agent, and see what happens.  You “challenge” them with heavy exposure to the disease, typically one that would otherwise guarantee infection in an un-vaccinated individual.

With that approach, there’s none of this waiting around for nature to take its course.  That’s more of a slam-bang, count the bodies and be done with it approach.  At some significant risk to the participants. (Although, the writeup above suggests that it would still take months to set up and run, which I find hard to believe if it were done on an emergency basis.)

Challenge trials are not a new idea.  They’re not even a new idea for testing COVID-19 vaccines.  Even some of Our Statesmen in Congress figured this out, months ago.

But, interestingly, Fauci appears to dismiss that possibility completely.  Meanwhile, “government scientists” (not otherwise identified in the news reporting) are creating the batches of COVID-19 that will only be needed if there are challenge trials.  I wonder which one is right.  And I wonder who the unnamed “government scientists” are.

But for now, I’m dismissing that possibility.  If they were doing to do challenge trials, they should have done them months ago.

Approximate statistical power calculation

In Post #774, I showed some of basic arithmetic that explains why it is so hard to test a COVID-19 vaccine.

  • It’s a deadly disease, so you can’t just take a few hundred people, give half a placebo, and expose them to COVID-19.  (That’s a challenge trial, as discussed above.)
  • Instead, you have to take tens of thousands of people, give half of them a placebo, and wait to see who gets infected in the normal course of business.
  • Given the low rate of new infections, it takes considerable person-months of time to accumulate enough infections to provide usable data.

I did the arithmetic in that prior post.  The US vaccine is aiming for a 30,000 person clinical trial.   For the sake of argument, let’s assume that they instantly enrolled all 30,000 people.  If they split that 50/50 (vaccine/placebo, good enough to make this point), at Virginia’s current infection rate (12 new infections per 100,000 population per day), you’d only accumulate about 50 infections per month in the placebo group. 

But that might vary quite a bit, just by chance.  Might be 20, might be 100.  And if the vaccine works, you’d accumulate fewer than 50/month, on average, for the vaccine group.  But that also would vary quite a bit, just by chance.  And then, based on the small difference between those small counts, you’d have to decide how well you think the vaccine is working.

That is why the people conducting vaccine clinical trials:

  • Chase down COVID-19 hotspots and recruit people there.  That gives them more infections per person-month of exposure.
  • Use huge samples (30,000 persons).  That gives them more stable numbers (more “statistical power”) for a given infection rate.

The final thing you need to know is that the FDA set a floor of 50% effectiveness, which they defined as either preventing infection or reducing severity of infection in half of cases.  A vaccine must be proven to do at least that well before they’ll approve it.

As an aside, that “or reducing severity of infection” may end up being weasel-wording of the highest order.  I know how to count infected versus not.  That’s black and white.  But as far as I can tell,  there’s literally no one legitimate way to calculate that second part about having reduced the severity of infection in X% of infected cases.   Manufacturers will have to establish an arbitrary scale of severity (e.g., death = 1, hospitalized on vent = 2, hospitalized no vent = 3, symptoms but not hospitalized = 4, asymptomatic case = 5).  And then perform some arithmetic on the counts of cases in each category, and from that conclude that they reduced the severity of infection in X% of cases.  Near as I can figure, that last part is going to be some form of hand-waving.  It will be some arbitrary method, applied to that arbitrary scale.  That opens the opportunities for presenting your data in the most favorable light possible.  To put it nicely.

In any event, the FDA 50% floor gives us the “target” for any statistical test.  The clinical trial is going to come up with an estimate of effectiveness, and an estimate of how uncertain your are about that number, the so-called “95% confidence interval”.  In everyday press, you’d see that expressed as a plus-or-minus range.  (E.g., the estimate is 60% effective, plus or minus 3%.  That would mean that, based on the data, you’re 95% sure it lies between 57% and 63%.  And if you repeated that clinical trial many times, only one time in 20 would you see numbers like that, but the true (real) effectiveness of the vaccine was somewhere outside of that range.)

There’s also a little statistical cheat you can try to use here, called a “one tailed test”.  It would be entirely inappropriate in this case (because a priori, you don’t know if you’re going to beat that 50% threshold or not), and I assume that legitimate scientists at FDA would not accept that.  So I’m basing this on a standard two-tailed test.

With all that, and putting aside the part about reducing severity, it’s easy enough to mock up the resulting statistical test.  To see just how much data you would need, in terms of person-days of exposure, to be able to exclude 50% effectiveness from the 95% confidence interval.  Assuming you were able to perform you test in a place with, say 20 infections / 100K persons/ day.

So that’s my rough back-of-the-envelope word problem.  Suppose you have a clinical trial with:

  • 15,000 enrolled
  • in an area with 20 infections/100K/day
  • split 50/50 into vaccine and placebo groups.  (The 50/50 split isn’t optimal, but this is just a rough calculation).

For a given true vaccine effectiveness of Y%, roughly how many days does that trial have to go on before you can exclude 50% effectiveness from the 95% confidence interval around your estimate of Y?  In other words, how long until the vaccine passes the test?

Details that nobody cares about but me:  I’m also going to use normal theory approximations here, because I know how to set up the problem in a spreadsheet that way.  I don’t think that matters, as long as I have well over 30 infections in both placebo and vaccination groups. And I’m modeling the FDA threshold as a known number, when in fact, it’s going to be half the placebo group number, which is itself uncertain.   I’m also ignoring the fact that the test is expressed as a ratio, which can complicate the statistics when both numerator and denominator are estimates (not known numbers). Finally, I’m ignoring that the trial doesn’t really start until the second month, because the US vaccine requires two doses, one month apart.

I built up this calculation using the standard formula for the variance of a binomial (yes/no) variable.  Then I use the normal approximation and, in effect, calculate a set of standard t-tests.  As mentioned above, that’s one of many things that’s not precisely correct about this calculation.  But it should be close enough.

Results are given below.  It’s no surprise that the better the vaccine, the less time it takes to prove it.  (To exclude 50% effectiveness from the 95% confidence interval).  What is surprising, to me, is how much overkill a 30,000 person trial is, if you expect the vaccine to be at all effective.  I think this may explain why other vaccines, such as the British (Jenner Institute/Astrazeneca) vaccine are using more like 10,000 in their clinical trials.

As an extras for experts, some of those very short intervals would provide too few infections to allow legitimate use of the “normal approximation” that I used here.  So those almost certainly overstate how short the trials would have to be, at the very bottom of the table.  The rule of thumb is, you’d want to see at least 30 infections in at least one of the groups.  At the placebo rate, then, that sets a floor of about three weeks to satisfy that additional constraint.  But the gist of this is still correct.  You only need a half-year of clinical trials if your vaccine doesn’t do much.

Post #792: Mask use and asymptomatic cases

I just stumbled across this article (or, same thing, here) and thought it was well worth repeating.  If for no other reason that a) it’s about COVID-19, b) it’s incredibly logical and fact-based, and c) it explains something fairly important.

The gist of this is that:

  1. When you are exposed to COVID-19, the amount of COVID-19 that you breathe in is a strong determinant of how sick you’ll get.  The higher the initial dose, all things equal, the sicker you’ll get.  The lower the dose, the less sick.
  2. Masks greatly reduce the amount of virus you inhale, when you are exposed.
  3. The least sick you can get is an “asymptomatic case”, that is, a person who was infected but fought back the virus without having any symptoms.
  4. A high rate of mask use greatly increases the fraction of infections that are asymptomatic infections.

Continue reading Post #792: Mask use and asymptomatic cases

Post #790: Cigarette smoke does not work as a test for mask filtration ability

Way back in Post #750, 7/9/2020, I had the notion to use cigarette smoke to test the ability of masks to filter out aerosol-sized particles.  There is a need for some sort of home test, because it’s next-to-impossible to tell how well or poorly any off-the-shelf mask works.  That’s particularly true for the “KN95” masks now being sold in everywhere (Post #747).

The theory seemed sound.  Cigarette smoke particles are about the right size, and in the past, some people did in fact use N95 masks to try to avoid second-hand smoke.

Now, having executed this test on my back porch this afternoon, I can attest that it doesn’t work at all. I can smell cigarette smoke strongly right through a genuine (but quite old) 3M N95 respirator (upper left, above).  And I could not tell that the smell of smoke was any stronger when I used a worn-out 3M N95 dust mask (next), or a dust/surgical mask (blue) with no aerosol filtration capability. Continue reading Post #790: Cigarette smoke does not work as a test for mask filtration ability

Post #G24: Paw paw neurotoxicity.

Paw paws.  Source:  My yard.  Destination:  Recycle bin.

We have a couple of paw paw trees in our yard.  The are nice-looking trees, with large glossy green leaves.  I have the vague recollection that we put them in for butterfly habitat, as they are critical for the reproduction of the zebra swallowtail.

We rarely get any edible fruit from them, as the fruit always seem to go from rock hard to “the deer got them” in a matter of days.

And, as it turns out, that may have been a lucky break. Continue reading Post #G24: Paw paw neurotoxicity.

Post #750: Science alert: Proposed cigarette smoke test of “KN95” and other masks

Edit:  This did not work at all.  Not even a little bit.  See Post #790 for details.

In this post, I propose to rate masks on their ability to filter out cigarette smoke.  The particles in cigarette smoke are roughly the same size as virus/aerosol particles.  (Although their chemistry is quite different.)  I am going to use those second-hand smoke particles as my best proxy for human aerosol emissions. The basic idea being that if a particle-filtering mask is good at removing something as small as cigarette smoke particles, it probably does a pretty good job at filtering aerosols in general.

I’m announcing the full scope of the test ahead of time because that’s good science.  This way, I can’t just bury the results if they turn out unfavorable.  (AKA, toss them in the circular file.)  In some sense, it’s as important to know this doesn’t work, as it is to know that it does work, as a way to test masks.

The test is pretty straightforward:  Light a cigarette, hold it under your face, and rate how strongly you can smell cigarette smoke while breathing through the mask.  Less is better.

A cigarette is ideal for several reasons.  One, it provides a consistent burn, meaning, a consistent concentration of smoke particles from one mask to the next.  Two, it’s a readily-available and easily-repeatable standard.  Three, the smoke has been well-characterized.  Four, there is no risk (as with incense) that the material has been doused with molecules meant to volatilize as the base product is burned.  If you smell tobacco smoke, you are smelling smoke particles.  if you smell incense, it’s not clear what you are smelling.

I’m going to start with these five scenarios, hoping to establish some sort of scale:

  1. 3M N95 respirator (new old stock, should be true N95 filtration)
  2. Dust mask with two layers of Filtrete 1900 fabric (~ N85)
  3. Dust Mask with one layer of Filtrete 1900 fabric (~N60)
  4. Plain dust/surgical mask (N low).
  5. No mask (N00)

The first problem is, I still have to manufacture items 2 and 3.  I did a bunch of #3 masks for friends, early in on this pandemic.  I need to make some more, document that, and then go on to make #2 masks.  These should provide known filtration standards below N95.

Then I’ll rate the following six masks on that scale.  These are the masks whose performance I am trying to judge.  Starting with the beat-to-heck 3M dust mask that I have been wearing since the start of the pandemic, that I hope is still working well.  And then some other alternatives that are readily available to the US public.

  1. 3M N95 dust mask (extremely well-used)
  2. “KN95” mask #1, from Twins Ace Hardware in Fairfax
  3. “KN95” Mask  #2, from Twins Ace Hardware in Fairfax
  4. Generic single-use “surgical-style” mask #1.
  5. Generic single-use “surgical-style” mask #2.
  6. Plain-vanilla single layer cloth mask.

The point is to say whether or not you would materially improve your protection from aerosol-sized particles by swapping a plain-vanilla cloth mask for a typical  generic, non-certified “KN95” mask offered as an impulse item at our local Ace Hardware. Continue reading Post #750: Science alert: Proposed cigarette smoke test of “KN95” and other masks

Post #747: Can Kents clarify KN95 chaos? Updated

Source:  Depositphotos.com

Update 2:  This didn’t work, at all.  Not even a little bit.  See Post #790 for details.  You can’t use the odor of cigarette smoke to test mask filtration.

Update:  See postscript at bottom.  The ability of genuine N95 masks to filter smoke particles is well known and well documented.  In that light, my proposed “sniff test” for KN95 masks looks fairly promising.  To the extent that a mask reduces the odor of cigarette smoke, then it is filtering out virus-sized particles.

In Post #740, I noted that my local convenience store had “KN95” masks for sale.    I’ve heard a rumor that one of the local hardware stores is also selling such masks.  (I plan to check that out soon.)  And I exchanged emails with  neighbor who is in the process of purchasing some KN95s, from a couple of different sources, for daily wear at work.

In theory, wearing a KN95 gives you the same protection as an N95 respirator.  So, in theory, upgrading from a cloth mask or similar to a KN95 is a smart thing to do.

In practice, not so fast.  I’ve started looking into the “KN95” mask market, and it is complete chaos.  I guess that’s no surprise.  That’s more-or-less of a piece with the entire Federal response to COVID-19. Continue reading Post #747: Can Kents clarify KN95 chaos? Updated

Post #735: Flu and flu vaccine, part 1. Flu hardly matters.

Source:  US Centers for Disease Control.

The upshot of this posting is that flu doesn’t much matter, in terms of crowding of hospitals during the COVID-19 pandemic. When flu season comes around this year, if we’re still in the COVID-19 pandemic, the flu will add a bit of stress to the hospital system.  But only a bit.

Best guess, based on a variety of sources, the impact of the peak of a bad flu season, on hospital inpatient resources in any one state, will be maybe 5% of the size of the impact of the peak of the COVID-19 outbreak.  So far.  That’s based on number of hospital admissions, concentration of those admissions within states, and average hospital resource use per admission.  A bad flu season certainly won’t help things.  But it’s not really a make-or-break issue in this context.

In terms of things we need to worry about, I’m putting the influence of seasonal flu far, far down my list.  That, no matter what offhand remarks I may hear by public health officials about the dangers of a simultaneous COVID-19 and flu season.

Detail follows.

Continue reading Post #735: Flu and flu vaccine, part 1. Flu hardly matters.

Post #676: I paid my taxes yesterday

Which would not ordinarily be a thing to blog about.  So it’s an odd commentary on the times when doing my taxes was a pleasant change in routine.

I owed a bunch of money to the Commonwealth this year.  I figured that the Governor would appreciate receiving it in a timely fashion, all things considered.  So I did my taxes yesterday.

The deadline for Virginia is June 1 this year.  The Commonwealth says that if you pay by June 1, no penalties or interest are due.  (My tax software said differently, so I’m not sure what the deal is.)

The Federal deadline, by contrast, is July 15.

So this year your Virginia income tax is due before your Federal.  But you have to fill out your Federal forms first, anyway, in order to do the Virginia form.  In terms of the time and hassle cost, tax day here in the Commonwealth is effectively June 1, no matter what the Feds say.

This was my first tax year in pure retiree mode, and boy was it an eye-opener. 

I ran my own little business here in Vienna until August 2018.  And I paid full freight, tax-wise.  As a self-employed person, you start off by paying both halves of Social Security and Medicare.  So that’s about 16% off the top.  (You get a bit of that back, as a deduction.)  And then you pay the rest.  The Town got its slice (Gross Receipts tax), then the Commonwealth, then the Feds.

To a pretty close approximation, my combined marginal tax rate used to be above 50%, and my average tax rate was about 33%.  That’s with stuffing as much as I legally could in an SEP-IRA.

To which my daughter would say, first-world problem.  Because it meant that I had a good job and my business wasn’t going bust.

But now?  No wage income means no social security.  Investment income means many of those lovely tax dodges set up for rich people now apply to me.  Add in some reduction in income, toss with progressive tax rates.  And voila:  I ended paying an average Federal rate of about 9%, and a marginal rate just slightly higher than that.

Which is, oddly enough, how I ended up owing so much to the state.  Back in the day (meaning, when I was working), Virginia taxes weren’t exactly rounding error, but compared to the Federal bite, they looked pretty darned reasonable.

And so, when I figured total taxes for withholding, I did my Federal, and used a rule-of-thumb for State.  Which is now wrong, because I now get all the geezer-related tax breaks from the Feds, but nothing like that from the Commonwealth.

Weirdly, the Commonwealth’s tax bite is the same as it always ways.  But now that my Federal rate is so ludicrously low, I’m kind of resenting the Virginia rate.

In the end, good sense prevails.  I’ve always thought I got excellent value for my Virginia tax dollar.  I’m not going to change my mind on that, now that I’m retired.  As stated, I suspect that the Commonwealth needs my money right about now.  So I got ’em done.

 

Post #618: Blue skies, a followup

White Clouds in Blue Sky ca. 1996

My wife found the definitive article in the Washington Post.  I’m not crazy, the air is significantly cleaner now, thanks to lockdown.

That article also has links to research suggesting that long-term exposure to “PM2.5”-type air pollution (fine particulates) explains much of the variation in coronavirus death rates across the country.

As I noted in an earlier post, Italian research points vaguely in that same direction.  Wuhan had notoriously bad air pollution, as did the hardest-hit region of Italy (the Po Valley).  And air quality in New York is not so good.  And, to be honest, that doesn’t bode well for DC.

So the sky really is better-looking these days.  And if the Italian analysis is right, the reduction in particulates helps slow the spread of disease.  But our long-term exposure to particulates likely increases the mortality rate among those who fall ill.