New research led by data science experts at the University of Illinois Urbana-Champaign and United Nations Global Pulse found that there is no strong evidence that YouTube promoted anti-vaccine sentiment during the COVID-19 pandemic.

The study, published in the Journal of Medical Internet Research, performed an algorithmic audit to examine if YouTube’s recommendation system acted as a “rabbit hole,” leading users searching for vaccine-related videos to anti-vaccine content.

For the study, the researchers asked World Health Organization-trained participants and workers from Amazon Mechanical Turk to intentionally find an anti-vaccine video with as few clicks as possible, starting from an initial informational COVID-19 video posted by the WHO.

They compared the recommendations seen by these users to related videos that are obtained from the YouTube application programming interface and to YouTube’s Up-Next recommended videos that were seen by clean browsers without any user-identifying cookies.

The team then used machine learning methods to classify anti-vaccine content, analyzing more than 27,000 video recommendations made by YouTube.

“We found no evidence that YouTube promotes anti-vaccine content to its users,” said Margaret Yee Man Ng, an Illinois journalism professor with an appointment in the Institute of Communications Research and lead author of the study. “The average share of anti-vaccine or vaccine hesitancy videos remained below 6% at all steps in users’ recommendation trajectories.”

The initial goal of the research was to better understand YouTube’s famously opaque techniques for content recommendations—going beyond querying the platform’s application programming interfaces to collect real-world data—and whether these techniques funnel users toward anti-vaccine sentiment and vaccine hesitancy.

“We wanted to learn about how different entities were using the platform to disseminate their content so that we could develop recommendations for how YouTube could do a better job of not pushing misinformation,” said UN Global Pulse researcher Katherine Hoffmann Pham, a co-author of the study. “Contrary to public belief, YouTube wasn’t promoting anti-vaccine content. The study reveals that YouTube’s algorithms instead recommended other health-related content that was not explicitly related to vaccination.”

“The videos that users were directed to were longer and contained more popular content, and attempted to push a blockbuster strategy to engage users by promoting other reliably successful content across the platform,” Ng said.

The study also allowed the researchers to examine how users’ real-world experiences differ from the personalized recommendations obtained by querying YouTube’s “RelatedToVideo” application programming interface. This API is designed to help programmers search for related content on the platform or using clean browsers, replicating the experience of a new user visiting YouTube with no search or view history, which is often used to study the platform’s recommendation system.

The study reports that the watch histories of users significantly affect video recommendations, suggesting that data from the API or a clean browser does not offer an accurate picture of the suggestions that real users are seeing. Real users saw slightly more pro-vaccine content as they advanced through their recommendation trajectories. In contrast, searches performed by the API or clean browsers during the study were drawn toward irrelevant recommendations as they advanced.

“I think one benefit of this study relative to others is that it proposes a relatively lightweight methodology to gather real data on how people navigate through YouTube’s video recommendations,” Pham said. “So unlike the APIs, which will just sort of randomly suggest new links, the users can critically review the links and pick one, which sort of mimics the behavior that many people would use on YouTube in reality.”

Understanding recommendation systems is important because it promotes transparency and holds them accountable,” added co-author Miguel Luengo-Oroz, a professor at the Telecommunications School of the Universidad Politecnica de Madrid. “This helps people understand the choices being made for them by platform designers.”