Infodemic Signal Detection During the COVID-19 Pandemic: Development of a Methodology for Identifying Potential Information Voids in Online Conversations

"As a contribution to the infodemic response toolbox, the taxonomy and methodology in this study offer a practical, structured approach for identifying information voids and narratives of concern that warrant attention and action."

Health emergencies give rise to information overload, including false or misleading information, in digital and physical environments. This "infodemic" can diminish trust in health authorities and threaten the public health response. Thus, most emergency and outbreak recommendations emphasise the value of listening to communities, involving them early in the response, and communicating clearly with them in a timely manner. This paper offers a methodology for social listening in a crisis context. Led by the World Health Organization (WHO) Information Network for Epidemics (EPI-WIN), the research proposes a way to identify and characterise points of confusion, harmful narratives, and key questions that signal information voids in the online social media space. Applying this approach on a weekly basis, as was done during the COVID-19 pandemic and could be repeated in future health emergencies, is meant to generate actionable insights to inform decision-making for a more effective response, including adapting risk communication and community engagement (RCCE) strategies.

The researchers developed a taxonomy to filter global public conversations in English and French related to COVID-19 on social media into 5 categories with 35 subcategories, with the top 5 being: (i) the cause of the disease; (ii) the illness; (iii) the treatment (e.g., vaccines); (iv) the interventions; (v) information (i.e., metaconversations about evidence and statistics, mis- and disinformation, successful and harmful content, or key influencers who have been actively amplifying information on COVID-19).  The taxonomy and its implementation were validated, and they were reviewed and adapted as language about the pandemic in online conversations changed over time.

The analysis was based on the weekly aggregation of publicly available social media data in English and French using Meltwater Explore; 87.02% of the resulting analysis data set was sourced from Twitter. The aggregated data for each subcategory were analysed on a weekly basis starting on March 23 2020 to detect signals of information voids based on 3 parameters: the volume (i.e., how many social media items referred to topic X?), the velocity (i.e., the rate of increase of the number of social media items that have engaged with topic X over the course of the past week), and the presence of questions about the topic. An analyst reviewed and identified potential information voids and sources of confusion, and quantitative data were used to provide insights on emerging narratives, influencers, and public reactions to COVID-19-related topics.

The identified issues, up to 10 topics, on social media were then further evaluated using engagement data and Google search trends to determine whether a significant number of online users had also been looking for information on these topics to help determine whether the information void was more widespread. The quantitative data were compiled in a web-based dashboard accessible to the emergency responders in the EPI-WIN team, and insights were discussed with EPI-WIN emergency responders on a weekly basis. The weekly updating of the dashboard enabled responders to see how online conversations shifted as the pandemic evolved over the course of 2020 and into 2021. (By May 4 2021, the data sample consisted of 1.02 billion unique social media posts.)

A key takeaway from the analysis is the frequent recurrence of topics of concern and its implications for communication. Public health authorities, governments, and non-governmental organisations (NGOs) must be prepared to communicate repeatedly on the same issue - adapting frames, approaches, and content as public perceptions of issues and topics shift.

As noted here, the taxonomy has been adapted, translated, and applied in country-level studies in Mali, the Philippines, and Malaysia. Applying the approach at the country level included the localisation of keywords and their validation. The taxonomy is also applicable at the subnational level: The research framework is being applied in Canada by the National Institute of Public Health of Québec as an input into the public health response and risk communication in that province. Furthermore, a pilot project by EPI-WIN and research partner Early Artificial intelligence-supported Response with Social Listening (EARS) built on the taxonomy and applied it to an automated classification of content and analysis of publicly shared opinions and concerns in 20 countries.

"[M]ore investment in analytical capacities in social listening at the country level is needed to provide more contextual analysis, interpretation of infodemic insights, and formulation of recommendations for action, as well as to build capacities for using social listening for health response evaluation and adaptation."

In conclusion: "The application of the taxonomy and methodology for social listening at regional, country, and subnational levels in the COVID-19 pandemic...offers possibilities for more actionable insights that must increasingly support a localized response. Moreover, this method offers an approach for monitoring of concerns, questions, and information voids in future outbreaks, enabling a faster response by the health authorities in affected countries during the next acute health event."