Assessing Message Deployment During Public Health Emergencies Through Social Media: Empirical Test of Optimizing Content for Effective Dissemination

"Message testing can be a vital tool to help communicators counter misinformation..."

During an infodemic, timely, reliable, and accessible information is crucial to combat the proliferation of health misinformation. While message testing can provide vital information to make data-informed decisions, traditional methods tend to be time- and resource-intensive. In response, the researchers developed the rapid message testing at scale (RMTS) approach to allow communicators to repurpose existing social media advertising tools and understand the full spectrum of audience engagement. This study is part of a larger vaccine confidence project conducted in Odisha, India.

Due to the low barrier to entry and potential for immediate and widespread dissemination, social media is often viewed as a cost-effective vehicle for a communication campaign, especially when time and reach are of the essence, such as during an infodemic. Platforms such as Google and Meta have created user-friendly interfaces that make low-cost digital advertising accessible to almost anyone. However, the number of available platforms - each with its own features and formats - makes designing messages quite complex. For example, the format may impact whether the video is watched at all, viewed for a few seconds or longer, or results in repeat viewing. With the RMTS approach, the researchers sought to develop a model of nuanced metrics based on a continuum of viewing experience so that communicators can make an evidence-based decisions before investing significant resources.

For this study, the researchers created 48 paid video advertisements on Facebook. The videos were all variations of one of the videos they had produced to promote uptake of the second dose of the COVID-19 vaccine. (See Related Summaries, below.) The original video length was 1 minute and 55 seconds and used narrative storytelling to address vaccine hesitancy. It features 1 protagonist (female) and uses a collectivist appeal. The video promotes the importance of being fully vaccinated against COVID-19 and includes a link that the viewer can click on to learn more and schedule an appointment to be vaccinated. For this study, the researchers created a shorter 58-second version of the video. For both the short and long versions of the video, they created a version that included subtitles in Odia, the language spoken in the video. The subtitles present the dialogue in text so that users who have their audio turned off, which is a popular behaviour for users consuming content on their phones, can also follow the dialogue. They then took each of the 4 video variations (long or short, with or without subtitles) and configured them in 3 different aspect ratios: 16:9 (rectangle), 1:1 (square), and 4:5 (portrait).

The videos were tested across 4 demographic groups (women or men, younger or older). The researchers assessed user engagement along a continuum of viewing experience: obtaining attention, sustaining attention, conveying the message, and inspiring action. These variables were measured by the percentage of video watched and the behavioural outcome or action, which takes the form of a click on the "learn more" button.

The video advertisements were placed on Facebook for over 4 consecutive days at a cost of US$450 and garnered 3.34 million impressions. Overall, the best-performing video was the shorter version in portrait aspect ratio and without subtitles. (The preference for portrait orientation can be explained by the fact that most users access Facebook on their smartphones, and most smartphone users hold their phones upright.) There was a significant but small association between the length of the video and users' level of engagement at key points along the continuum of viewing experience (N=1,032,888. For the longer video, those with subtitles held viewers longer after 25% video watch time than those without subtitles (n=15,597). While the researchers found some significant associations between the aspect ratio, the use of subtitles, and the number of users watching the video and clicking on the call-to-action link, the effect sizes were extremely small.

Thus, the data suggest that on Facebook, shorter videos (<1 minute) shown in portrait orientation and without subtitles can lead to greater viewer engagement but that subtitles may be beneficial in retaining views on longer videos (>1 minute). These findings may inform the construction of videos for dissemination on social media - editing the video for a shorter or perhaps longer beginning if, for example, the goal is to keep attention for as long as possible. 

Based on the study, the researchers offer several directions for future research. For example, the content of the video itself was developed in collaboration with a Community Advisory Board in Odisha, and so, in a way, could be considered culturally specific. Future research could explore the impact that cultural differences have on message reception and engagement, especially given the global nature of social media platforms. Future studies could also look at incorporating other forms of content (beyond video advertisements) and applying the RMTS approach using other social media platforms. Notably, special care needs to be taken when conducting tests during an infodemic, where there is already an overwhelming amount of information, some of which may be mis- or disinformation, competing for users' attention and thus potentially diluting the efforts of public health agencies.

In conclusion, this test served as a proof of concept for the RMTS approach. The researchers obtained rapid feedback on formal message attributes from a very large sample. The results of this test reinforce the need for platform-specific tailoring of communications: "RMTS testing highlights nuances that communication professionals can address instead of being limited to a 'one size fits all' approach."