When virologist José Esparza began working with the World Health Organization to combat the AIDS epidemic in the 1980s, he and many of his colleagues were convinced that a vaccine would be the solution – and that it would come quickly.

Their optimism rested on solid science: Researchers knew that people produce antibodies to the human immunodeficiency virus that causes AIDS. And spurring the body to produce antibodies was already a common and successful vaccine strategy that had drastically reduced cases of measles, smallpox and many other diseases. Tackling AIDS seemed equally doable.

“We thought that it was going to be a piece of cake,” says Esparza, a former senior advisor at the Bill & Melinda Gates Foundation, who is now affiliated with the University of Maryland School of Medicine. “We didn’t know the complexity of HIV.”

More than three decades later, there is still no viable vaccine candidate for HIV, even as scientists have produced multiple effective vaccines for the SARS-CoV-2 virus that causes Covid-19 less than a year after it emerged.

Now, recent findings are sparking new hope. At an international AIDS conference in February, researchers from Scripps Research and IAVI, a non-profit vaccine research organisation, announced promising blood-test results from the first phase one human trial of a new HIV vaccine strategy.

The results, which are still unpublished, grabbed the public’s attention in a way that could only be possible in the era of social media. “THIS IS HUGE,” read a tweet by Twitter user @AugustusRotter. The tweet was liked and retweeted many thousands of times in early April.

The reality is far more nuanced than the hype suggests, says William Schief, an immunologist at Scripps and executive director of vaccine design at IAVI’s Neutralizing Antibody Center. Although the immune response his team detected is an important proof-of-concept finding, he says, the field is still years away from producing shots that make people less likely to be infected with HIV.

Even then, an eventual vaccine will likely include multiple shots, which may be a tough sell.

“Scientifically, it is a beautiful concept,” Esparza says. “Practically, it will not be easy to implement.”

Still, after decades of setbacks, the results are welcome news – with some intriguing connections to the Covid-19 vaccine effort that may help accelerate the HIV work.

“It’s kind of a small step toward making an HIV vaccine, but it’s also a gigantic step,” suggesting a viable path forward, Schief says. “And in fact, in this particular case, it worked amazingly well.”

The search for an HIV vaccine began soon after scientists isolated the virus and confirmed that it caused AIDS in 1984. Since then, the scientific quest has followed three waves of research, says Esparza, who published a historical account of the search for an HIV vaccine in 2013.

The first wave focused on the most established idea: trying to stimulate the human immune system to produce so-called neutralising antibodies, which inactivate specific viruses. This is the strategy that many other vaccines use, including those for Covid-19. For years, researchers worked to identify the antibodies that people produced as they responded to HIV infection, and then develop vaccines that would induce the production of similar antibodies.

But HIV turned out to be an elusive enemy. Antibodies target specific proteins on the surface of a virus. However, HIV mutates rapidly into variants that the antibodies can’t recognize, which means it continuously stays one step ahead of the immune system.

In one classic study, Schief says, researchers repeatedly tested the blood of people infected with HIV and found that the antibodies produced by their immune systems were always about three to six months behind the virus.

“HIV is still a much tougher scientific target,” than SARS-CoV-2, says Larry Corey, an expert in virology, immunology and vaccine development at the Fred Hutchinson Cancer Research Center in Seattle, and principal investigator of the HIV Vaccine Trials Network. “Ninety-eight percent of humans recover from SARS-CoV-2, and we are zero out of 78 million of self-recovery for HIV.”

By the early 2000s, researchers were pursuing a second wave of HIV vaccines, based on the idea of targeting the body’s soldier-like “killer” T cells instead of trying to stimulate antibodies. Long-term human immunity depends on two main groups of cells: B cells and T cells. Both help produce antibodies, but T cells also seek out and destroy infected cells. The idea for T-cell vaccines was to stimulate cells that recognize internal proteins in the virus.

In 2007, that idea not only failed to provide protection in a double-blind, randomised phase two trial called STEP, it also appeared to increase the risk of HIV infection. “The trial failed miserably,” Esparza says.

That was far from the only vaccine attempt to fizzle. After decades of human trials, only one has shown any degree of real-world effectiveness. Completed in 2009 in Thailand, a combination of two vaccines, which took the wave-one strategy of inducing antibodies, reduced rates of HIV infection by 31 per cent – not enough to gain regulatory approval.

The third and current wave of HIV vaccine research began in the late 2000s, when researchers discovered that a small minority of people infected with HIV produce particularly potent antibodies that can neutralise many strains of HIV at once. So far, scientists have identified dozens of these broadly neutralising antibodies, which target parts of the viral surface (much like the spike proteins on SARS-CoV-2) that are consistent from strain to strain.

People who make these proteins still cannot fight off HIV because their bodies don’t make these antibodies until after the viral infection has taken hold and the virus continues to mutate in the meantime, Schief says.

But the discovery sparked a new idea: Maybe an effective vaccine could get a step ahead of the virus by targeting so-called naïve B cells (also known as precursor cells), which circulate in our blood, Schief says. If a vaccine could cause naïve B cells to acquire mutations that transform them into cells that produce broadly neutralising antibodies before an HIV infection, the body might be able to fight it off when presented with the virus for the first time.

• A National Geographic report