Business Analytics in Action: Confidence Intervals

Part I:

Please read NYTimes article What the Efficacy Numbers Mean, And How You Can Compare Them (Links to an external site.)Links to an external site. ... (click here for article in pdf formatDownload click here for article in pdf format)

Briefly (paragraph or two) explain what you learned and how it relates to topics we've covered in this class. What does it mean to you?

Page

of 3

ZOOM

What Do Vaccine Efficacy Numbers Actually Mean? Zimmer, Carl; Collins, Keith. New York Times (Online), New York: New York Times Company. Mar 3, 2021. This week, Johnson & Johnson began delivering millions of doses of its coronavirus vaccine across the United States after receiving an emergency use authorization from the Food and Drug Administration. Central to getting the green light was a trial that Johnson & Johnson ran to measure the vaccines efficacy. Efficacy is a crucial concept in vaccine trials, but its also a tricky one. If a vaccine has an efficacy of, say, 95 percent, that doesnt mean that 5 percent of people who receive that vaccine will get Covid-19. And just because one vaccine ends up with a higher efficacy estimate than another in trials doesnt necessarily mean its superior. Heres why. For statisticians, efficacy is a measurement of how much a vaccine lowers the risk of an outcome. For example, Johnson & Johnson observed how many people who received a vaccine nevertheless got Covid-19. Then they compared that to how many people contracted Covid-19 after receiving a placebo. The difference in risk can be calculated as a percentage. Zero percent means that vaccinated people are at as much risk as people who got the placebo. A hundred percent means that the risk was entirely eliminated by the vaccine. In the United States trial site, Johnson & Johnson determined that the efficacy is 72 percent. Efficacy depends on the details of a trial, such as where it took place. Johnson & Johnson ran trials at three sites: in the United States, Latin America and South Africa. The overall efficacy was lower than that in the United States alone. One reason for that appears to be that the South Africa trial took place after a new variant had swept across that country. Called B.1.351, the variant has mutations that enable it to evade some of the antibodies produced by vaccination. The variant didnt make the vaccine useless, however. Far from it: In South Africa, Johnson & Johnsons efficacy was 64 percent. Efficacy can also change when scientists look at different outcomes. Johnson & Johnsons vaccine had an 85 percent efficacy rate against severe cases of Covid-19, for example. Thats important to know, because it means that the vaccine will prevent a lot of hospitalizations and deaths. When scientists say that a vaccine has an efficacy of, say, 72 percent, thats whats known as a point estimate. Its not a precise prediction for the general public, because trials can only look at a limited number of people in the case of Johnson & Johnsons trial, about 45,000 volunteers.

The uncertainty around a point estimate can be small or large. Scientists represent this uncertainty by calculating a range of possibilities, which they call a confidence interval. One way of thinking of a confidence interval is that we can be 95 percent confident that the efficacy falls somewhere inside it. If scientists came up with confidence intervals for 100 different samples using this method, the efficacy would fall inside the confidence intervals in 95 of them. Confidence intervals are tight for trials in which a lot of people get sick and theres a sharp difference between the outcomes in the vaccinated and placebo groups. If few people get sick and the differences are minor, then the confidence intervals can explode. Last year, the F.D.A. set a goal for coronavirus vaccine trials. Each manufacturer would need to demonstrate that a vaccine had an efficacy of at least 50 percent. The confidence interval would have to reach down no lower than 30 percent. A vaccine that met that standard would offer the kind of protection found in flu vaccines and would therefore save many lives. So far, three vaccines made by Pfizer and BioNTech, Moderna and Johnson & Johnson have all been authorized in the United States after their trials demonstrated they surpassed the F.D.A.s threshold. AstraZeneca and Novavax, which have ongoing U.S. trials, have published efficacy results from studies in other countries. Meanwhile, the makers of the Sputnik V vaccine have published results based on their trial in Russia. For a number of reasons, its not possible to make a precise comparison between these vaccines. One vaccine may have a higher point estimate than another, but their confidence intervals may overlap. That effectively makes their results indistinguishable. Making matters more complicated, the vaccines were tested on different groups of people at different stages in the pandemic. In addition, their efficacy was measured in different ways. Johnson & Johnsons efficacy was measured 28 days after a single dose, for example, while Modernas was measured 14 days after a second dose. Whats clear is that all three vaccines authorized in the United States made by Johnson & Johnson, Moderna, and Pfizer and BioNTech greatly reduce the risk of getting Covid-19. Whats more, all the vaccines look as if they have a high efficacy against more serious outcomes like hospitalization and death. For example, no one who got Johnson & Johnsons vaccine had to go to the hospital for a Covid-19 infection 28 days or more after getting an injection. Sixteen people who got the placebo did. That translates to 100 percent efficacy, with a confidence interval of 74.3 percent to 100 percent. A clinical trial is just the start of the research on any vaccine. Once it goes into widespread use, researchers follow its performance. Instead of efficacy, these scientists now measure effectiveness: how much the vaccine reduces the risk of a disease out in the real world, in millions of people rather than thousands. Early studies on the effectiveness of coronavirus vaccines are confirming that they provide strong protection.

In the months to come, researchers will keep an eye on this data to see if they become less effective either because the immunity from the vaccine wanes or because a new variant arises. In either case, new vaccines will be created, and manufacturers will provide new measures of their efficacy. Word count: 980 Copyright 2021 The New York Times Company