Advertisement

SKIP ADVERTISEMENT

Comments

What Do Vaccine Efficacy Numbers Actually Mean?Skip to Comments
The comments section is closed. To submit a letter to the editor for publication, write to letters@nytimes.com.

What Do Vaccine Efficacy Numbers Actually Mean?

This week, Johnson & Johnson began delivering millions of doses of its coronavirus vaccine across the United States after receiving an emergency use authorization from the Food and Drug Administration. Central to getting the green light was a trial that Johnson & Johnson ran to measure the vaccine’s efficacy.

Efficacy is a crucial concept in vaccine trials, but it’s also a tricky one. If a vaccine has an efficacy of, say, 95 percent, that doesn’t mean that 5 percent of people who receive that vaccine will get Covid-19. And just because one vaccine ends up with a higher efficacy estimate than another in trials doesn’t necessarily mean it’s superior. Here’s why.

Share of U.S. Johnson & Johnson vaccine trial volunteers who got Covid-19

Placebo group

Vaccinated group

63 per 5,000 volunteers

contracted Covid-19

18 per 5,000

72%

less risk

(estimated

efficacy)

Placebo group

Vaccinated group

63 per 5,000 volunteers

contracted Covid-19

18 per 5,000

72%

less risk

(estimated

efficacy)

Notes: Numbers are rounded. There were 8,835 volunteers in the placebo group, and 112 of them (1.27 percent) were infected with Covid-19; there were 8,958 volunteers in the vaccinated group, and 32 of them (0.36 percent) were infected.·Source: The Food and Drug Administration’s analysis of clinical trials conducted by Johnson & Johnson

For statisticians, efficacy is a measurement of how much a vaccine lowers the risk of an outcome. For example, Johnson & Johnson observed how many people who received a vaccine nevertheless got Covid-19. Then they compared that to how many people contracted Covid-19 after receiving a placebo.

The difference in risk can be calculated as a percentage. Zero percent means that vaccinated people are at as much risk as people who got the placebo. A hundred percent means that the risk was entirely eliminated by the vaccine. In the United States trial site, Johnson & Johnson determined that the efficacy is 72 percent.

Efficacy depends on the details of a trial, such as where it took place. Johnson & Johnson ran trials at three sites: in the United States, Latin America and South Africa. The overall efficacy was lower than that in the United States alone. One reason for that appears to be that the South Africa trial took place after a new variant had swept across that country. Called B.1.351, the variant has mutations that enable it to evade some of the antibodies produced by vaccination. The variant didn’t make the vaccine useless, however. Far from it: In South Africa, Johnson & Johnson’s efficacy was 64 percent.

Efficacy can also change when scientists look at different outcomes. Johnson & Johnson’s vaccine had an 85 percent efficacy rate against severe cases of Covid-19, for example. That’s important to know, because it means that the vaccine will prevent a lot of hospitalizations and deaths.

Share of worldwide Johnson & Johnson trial volunteers who got Covid-19

Placebo group

Vaccinated group

49 per 5,000 volunteers

contracted Covid-19

17 per 5,000

66%

less risk

(estimated

efficacy)

9 per 5,000 contracted

severe cases

1 per 5,000

85%

less risk of severe cases

Placebo group

Vaccinated group

49 per 5,000 volunteers

contracted Covid-19

17 per 5,000

66%

less risk

(estimated efficacy)

9 per 5,000 contracted

severe cases

1 per 5,000

85%

less risk of severe cases

Notes: Numbers are rounded. There were 19,544 volunteers in the placebo group: 193 of them (0.99 percent) were infected with Covid-19, and 34 (0.17 percent) were severely infected. There were 19,514 volunteers in the vaccinated group: 66 of them (0.34 percent) were infected, and 5 (0.03 percent) were severely infected.·Source: The Food and Drug Administration’s analysis of clinical trials conducted by Johnson & Johnson

When scientists say that a vaccine has an efficacy of, say, 72 percent, that’s what’s known as a point estimate. It’s not a precise prediction for the general public, because trials can only look at a limited number of people — in the case of Johnson & Johnson’s trial, about 45,000 volunteers.

The uncertainty around a point estimate can be small or large. Scientists represent this uncertainty by calculating a range of possibilities, which they call a confidence interval. One way of thinking of a confidence interval is that we can be 95 percent confident that the efficacy falls somewhere inside it. If scientists came up with confidence intervals for 100 different samples using this method, the efficacy would fall inside the confidence intervals in 95 of them.

Confidence intervals are tight for trials in which a lot of people get sick and there’s a sharp difference between the outcomes in the vaccinated and placebo groups. If few people get sick and the differences are minor, then the confidence intervals can explode.

Efficacy confidence intervals from major vaccine trials

Trials not conducted in the presence of widespread B.1.351 variant

Efficacy

estimate

Novavax

2 doses,

3 weeks apart

14,049 participants

in the U.K.

Lower

bound

Upper

bound

2 doses,

3 weeks apart

Gamaleya Research

Institute (Sputnik V)

19,866 participants in Russia

Moderna

2 doses,

4 weeks apart

27,817 participants

in the U.S.

Pfizer/BioNTech

2 doses,

3 weeks apart

36,523 participants in

Argentina, Brazil,

Germany, South Africa,

Turkey and the U.S.

40

60

80

100

20%

Trials conducted in the presence of widespread B.1.351

Novavax

2 doses,

3 weeks apart

2,684 participants in

South Africa

AstraZeneca/Oxford

2 doses,

4 weeks apart

8,895 participants in Brazil,

South Africa and the U.K.

Johnson & Johnson

1 dose

39,058 participants in Brazil,

South Africa and the U.S.

40

60

80

100

20%

Efficacy required by the F.D.A.

for authorization in the U.S.

Trials not conduced in the presence of widespread B.1.351 variant

Efficacy

estimate

Novavax

2 doses, 3 weeks apart

Lower

bound

Upper

bound

14,049 participants

in the U.K.

Gamaleya Research

Institute (Sputnik V)

2 doses, 3 weeks apart

19,866 participants in Russia

Moderna

2 doses, 4 weeks apart

27,817 participants

in the U.S.

Pfizer/BioNTech

2 doses, 3 weeks apart

36,523 participants in

Argentina, Brazil,

Germany, South Africa,

Turkey and the U.S.

20%

40

60

80

100

Trials conducted in the presence of widespread B.1.351

Novavax

2 doses, 3 weeks apart

2,684 participants in

South Africa

AstraZeneca/Oxford

2 doses, 4 weeks apart

8,895 participants in Brazil,

South Africa and the U.K.

Johnson & Johnson

1 dose

39,058 participants in Brazil,

South Africa and the U.S.

20%

40

60

80

100

Efficacy required by the F.D.A.

for authorization in the U.S.

Sources: The Food and Drug Administration’s analysis of clinical trials conducted by Pfizer, Johnson & Johnson and Moderna; clinical trial results published by AstraZeneca and Gamaleya Research Institute; company press releases.

Last year, the F.D.A. set a goal for coronavirus vaccine trials. Each manufacturer would need to demonstrate that a vaccine had an efficacy of at least 50 percent. The confidence interval would have to reach down no lower than 30 percent. A vaccine that met that standard would offer the kind of protection found in flu vaccines — and would therefore save many lives.

So far, three vaccines — made by Pfizer and BioNTech, Moderna and Johnson & Johnson — have all been authorized in the United States after their trials demonstrated they surpassed the F.D.A.’s threshold. AstraZeneca and Novavax, which have ongoing U.S. trials, have published efficacy results from studies in other countries. Meanwhile, the makers of the Sputnik V vaccine have published results based on their trial in Russia.

For a number of reasons, it’s not possible to make a precise comparison between these vaccines. One vaccine may have a higher point estimate than another, but their confidence intervals may overlap. That effectively makes their results indistinguishable.

Making matters more complicated, the vaccines were tested on different groups of people at different stages in the pandemic. In addition, their efficacy was measured in different ways. Johnson & Johnson’s efficacy was measured 28 days after a single dose, for example, while Moderna’s was measured 14 days after a second dose.

What’s clear is that all three vaccines authorized in the United States — made by Johnson & Johnson, Moderna, and Pfizer and BioNTech — greatly reduce the risk of getting Covid-19.

What’s more, all the vaccines look as if they have a high efficacy against more serious outcomes like hospitalization and death. For example, no one who got Johnson & Johnson’s vaccine had to go to the hospital for a Covid-19 infection 28 days or more after getting an injection. Sixteen people who got the placebo did. That translates to 100 percent efficacy, with a confidence interval of 74.3 percent to 100 percent.

A clinical trial is just the start of the research on any vaccine. Once it goes into widespread use, researchers follow its performance. Instead of efficacy, these scientists now measure effectiveness: how much the vaccine reduces the risk of a disease out in the real world, in millions of people rather than thousands. Early studies on the effectiveness of coronavirus vaccines are confirming that they provide strong protection.

In the months to come, researchers will keep an eye on this data to see if they become less effective — either because the immunity from the vaccine wanes or because a new variant arises. In either case, new vaccines will be created, and manufacturers will provide new measures of their efficacy.