Manufacturers of a third coronavirus vaccine announced positive results in clinical trials last week, setting off yet another round of excited news reports.
This one, produced by a partnership between a University of Oxford research institute, its spinout company Vaccitech and the pharmaceutical company AstraZeneca does not need to be stored at freezing temperatures and would be cheaper and easier to produce than the high-efficacy vaccines produced by BioNTech-Pfizer and Moderna.
Indeed, according to an initial media reports, Oxford-AstraZeneca’s is “expected to be relied upon heavily across the globe, to help curb a pandemic that has killed more than 1.3 million people.”
Sounds like great news. The press release from AstraZeneca presents “convincing evidence that [the vaccine] works,” said Science. But not everyone has been convinced. The price of AstraZeneca’s shares actually dropped on the news and an analysis from an investment bank concluded,
“We believe that this product will never be licensed in the US.” Over at STAT News, Anthony Fauci cautioned that we’ll need to see more data before coming to a conclusion. The sceptics have strong reasons to be concerned: This week’s “promising” results are nothing like the others that we’ve been hearing about in November – and the claims that have been drawn from them are based on very shaky science.
The problems start with the fact that the announcement did not present results from a single, large-scale, Phase Three clinical trial, as was the case for earlier bulletins about the BNT-Pfizer and Moderna vaccines. Instead, Oxford-AstraZeneca’s data came out of two separate studies: one in the UK that began in May and another in Brazil, which started at the end of June.
These two studies were substantially different from one another: They didn’t have standardized dosing schemes across the trials, for one thing, nor did they provide the same “control” injections to volunteers who were not getting the experimental Covid vaccine.
The fact that they may have had to combine data from two trials in order to get a strong enough result raises the first red flag.
Consider that leading vaccine-makers – including AstraZeneca – issued a scientific-rigour-and-integrity pledge back in September in which they promised to submit their products for approval or emergency use authorisation only “after demonstrating safety and efficacy through a Phase Three clinical study that is designed and conducted to meet requirements of expert regulatory authorities such as FDA.”
Note the wording here: These companies did not suggest that they might claim to have demonstrated efficacy through multiple, distinct clinical studies, combined together to get enough data. They said they would use a Phase Three study – as in, one big one.
Yet AstraZeneca has already applied on the basis of this data for approval in Canada and has plans to do the same in Britain, Europe and Brazil. The company also says it will use the data to apply for emergency use authorization in the US.
The Food and Drug Administration’s guidance for Covid-19 vaccines does allow for emergency use authorisation based on interim analyses, but the same document says this must be supported by a minimum level of vaccine efficacy “for a placebo-controlled efficacy trial.”
When BNT-Pfizer’s discussions with the FDA led to changes in this plan, BNT-Pfizer explained why, and released an updated protocol. That’s scientific rigor, and it matters a lot. When a vaccine-maker specifies the rules of the game before the results start coming in, we can check their work and be confident in what they tell us at the end. We can make sure they have not cherry-picked the data.
How did Oxford-AstraZeneca end up with this patched-together analysis instead of data from a single, large trial?
The Oxford-AstraZeneca story is very different, though. Presumably, neither of the two trials from which they combined data could have provided a clear answer on the vaccine’s efficacy on its own. To make things worse, Oxford-AstraZeneca reported only the results for certain subgroups of people within each one.
For perspective on this: The two subgroups chosen leave out perhaps half the people in the Brazilian trial.
Meanwhile, one of their key claims is that giving half a dose of the vaccine on the first injection, followed by a standard dose on the second one, led to better outcomes – but neither of these trials had been designed to test this hypothesis.
In fact, has since emerged that the half-dose/full-dose option started out as a mistake and one that was only caught when some people in the study did not have the usual high rate of adverse effects.
There were other dosing issues, too, that have not been explained even though dosing is the centrepiece of the press release. There are many different regimens in these trials – the UK study has more than two dozen arms, meaning the volunteers were divided into that many groups according to age and how much of the vaccine would be administered and when.
The doses are measured by the number of altered viral particles they contain and the developers decided that the standard dose would be 5×1010 viral particles. But for many of those arms in the UK trial – as well as everyone who got the vaccine in the Brazilian trial – publicly available trial information shows that the standard dose could be between 3.5 and 6.5×1010 viral particles. The lower end of that range is not far off from a half-dose.
How did Oxford-AstraZeneca end up with this patched-together analysis instead of data from a single, large trial? After all, this vaccine went into Phase Three testing before either BNT-PFizer’s or Moderna’s did. But in the UK, where that testing started, the Covid-19 outbreak happened to be receding. That meant results would be coming in very slowly.
In the UK, for example, the volunteers who did not get the experimental Covid vaccine were injected with meningococcal vaccine; in Brazil, those in the comparison group were given a saline injection as a placebo.
Meanwhile, BNT-Pfizer and Moderna began Phase Three trials for their coronavirus vaccines on the same day in July: Both planned to include 30,000 volunteers at the time and both trial plans were approved by the FDA. Oxford-AstraZeneca then announced they, too, would run a 30,000-person trial in the US.
But that research on the Oxford-AstraZeneca vaccine quickly fell behind the others. The US trial was approved by the FDA, but it did not start recruiting people until the end of August; and just a week later, it was put on hold so the FDA could investigate a serious adverse event in the UK trial. It was not clear what caused the volunteer to get sick, but the FDA did not give the all-clear for Oxford-AstraZeneca’s US trial to resume until October 23. By then the protocol for the trial had been publicly released.
So here we are at the end of November. BNT-Pfizer and Moderna have offered up a masterclass in how to do major vaccine trials quickly in a pandemic, while Oxford-AstraZeneca has, for the moment, only an assortment of smaller ones ready to look at.
Last week, Oxford-AstraZeneca published some results from earlier trials in the development of the UK trial. That paper included a trial protocol for the UK study, attached as an appendix.
Deep in that document, and apparently overlooked by reporters and commentators, was an eyebrow-raising suggestion: Under a section marked “Interim and primary analyses of the primary outcome,” the trialists outline a plan to combine and analyse data from four clinical trials (only half of which are Phase Three), carried out in different ways on three different continents.
The plan, they wrote, was to pull out results for people across these four trials and then pool them together for what is called a meta-analysis.
The appendix does not say when this became the plan. We don’t even know if the Oxford-AstraZeneca team followed it. In fact, it is impossible to know, at this point, just how many analyses these researchers have run and on which data. That is a scientific red flag with flashing lights.
Again, it is useful to compare this work to the BNT-Pfizer and Moderna trials, where the analyses were clearly spelt out ahead of time for everyone to see. All we know for sure is that last week, Oxford-AstraZeneca announced results of a different interim analysis that included only volunteers from the two trials in the UK and Brazil.
There are other problems, too. In the press release, Oxford-AstraZeneca reports that two of the dosing regimens “demonstrated efficacy.” Presumably, none of the others did, but they did not give specifics. Of the only two regimens they reported, one (the mistaken first half-dose, followed by a full dose at least a month later) came in at 90 per cent and the other (two standard doses at least a month apart) achieved only 62 percent efficacy. You’ll see reports that the vaccine had 70 per cent efficacy, on average; but that is unknowable, because we only have numbers on these two regimens.
As far as we know, some of this analysis could hinge on data from just a few sick people.
That is not the end of the problems. Overall, the Oxford-AstraZeneca trials appear to include relatively few participants over the age of 55, even though this group is especially vulnerable to Covid-19. People over 55 were not originally eligible to join the Brazilian trial at all.
– A Nature magazine report