Use of Community Engagement Interventions to Improve Child Immunisation in Low- and Middle-Income Countries: A Systematic Review and Meta-Analysis

"We found that community engagement features themselves were associated with intervention success. If the intervention involved holding stakeholder consultations, holding community dialogues, involving community leaders, those features were associated with intervention success." - Monica Jain, Lead Evaluation Specialist at 3ie

Community engagement interventions are increasingly being emphasised in international and national policy frameworks as a means to improve immunisation coverage and reach marginalised communities. This systematic review examines the effectiveness and cost-effectiveness of community engagement interventions on outcomes related to childhood immunisation in low- and middle-income countries (LMICs) and identifies contextual, design, and implementation features that may be associated with effectiveness. Specifically, the review aimed to answer the following four questions:

1. What evidence exists regarding the effectiveness of community engagement interventions in improving routine immunisation coverage of children in LMICs?
2. Is there evidence for heterogeneous effects of community engagement strategies (i.e., does effectiveness vary by region, population, gender, or programme implementation)?
3. What factors relating to programme design, implementation, context, and mechanism are associated with better or worse outcomes along the causal chain? Do these vary by the kind of community engagement?
4. What is the cost-effectiveness of different community engagement interventions in improving children routine immunisation outcomes?

For the purposes of the review:

• A community comprises a group of people who are served by a particular primary health facility (e.g., a subdistrict-level health centre). Thus, communities encompass a wide range of stakeholders, including caregivers, health service providers, and influential community members such as religious or other traditional leaders.
• Community participation is defined as a process wherein communities are included, to various degrees, in planning, decision-making, and implementation of activities that directly impact them. The researchers draw on the Spectrum of Public Participation, developed by the International Association for Public Participation (iap2.org), which identifies five levels of public (community) participation, ranging from informing communities to empowering them.
• The researchers developed a conceptual framework for various types of community engagement to assess the studies for inclusion. They consider three points within an intervention during which engagement can occur: in the design of the intervention, in the implementation of the intervention, or embedded in the intervention.

The initial searches were conducted in May 2019, and an updated search was done in May 2020 - returning over 46,000 papers. The review synthesises the results from 61 quantitative impact evaluations (IE), 47 qualitative studies, and 69 project reports that met inclusion criteria. The cost-effectiveness synthesis is based on 14 of the 61 IE that have the required combination of cost and effectiveness data. The studies were conducted in 19 LMICs all over the world, with a majority coming from South Asia and Sub-Saharan Africa.

The authors appraised the quality of both quantitative and qualitative, as well as cost, evidence base. The quantitative evidence was mostly low quality, though the randomised studies were generally of higher quality than quasi-experimental studies. Risk of bias assessments were conducted on the 47 included qualitative papers, and their quality was generally high. The quality of the cost evidence was mixed.

The majority of the studies largely fell into two engagement classifications, engagement as the intervention (engagement is embedded) (27) and engagement in design (16). There were 15 studies that fell into more than one engagement classification and were classified as having multiple engagement types.

Selected results:

• Review question 1: The evidence indicates that community engagement interventions had a small but significant positive effect on all the primary immunisation outcomes related to coverage and their timeliness. The findings are robust to exclusion of studies assessed as high risk of bias. The average pooled effect on full immunisation coverage in the random effects (RE) model was an increase of 0.14 standard deviations units (95% confidence interval [CI]: 0.06, 0.23) across all kinds of community engagement interventions. The community engagement interventions also had positive and mostly significant effect on the timeliness of vaccinations and led to an average pooled increase of 0.15 standard deviation units (95% CI: 0.07, 0. 24) in the timeliness of full immunisation.
• Review question 2: Among the types of community engagement interventions, it was the engagement as the intervention (engagement is embedded), which involves creation of community buy-in or development of new cadres of community-based structures, that had consistent positive effects on more primary vaccination coverage outcomes than the other engagement types. For example, it (engagement as the intervention) had a small but positive and significant increase in full immunisation coverage by 0.08 standard deviations (95% CI: 0.03, 0.13).
• Review question 3: The qualitative synthesis highlighted the importance of accounting for contextual factors that could act as barriers or facilitators to immunisation. For example, poor quality of services, including uninviting attitudes of health workers, posed a barrier to immunisation in communities that received engagement as the intervention or were engaged in the design of the intervention. Interventions that acknowledge or address these barriers are likely to be more effective in improving outcomes. Across all engagement types, most studies associated favourable social norms, caregivers' awareness and perception of the benefits of vaccination, and high maternal education rates to improved immunisation outcomes. In terms of programme design, certain aspects of community engagement itself, such as conducting stakeholder consultations, holding community dialogues, or involving community leaders, were associated with better immunisation outcomes. Among the studies that attributed intervention failure to intervention characteristics, inadequate duration frequency or exposure to the intervention were the most notable reasons. Studies reported that certain uptake and fidelity challenges may have resulted in inefficient mechanisms for change. For example, administrative challenges, particularly related to technical issues and communication, were common.
• Review question 4: The median intervention cost per treated child per vaccine dose (excluding the cost of vaccines) to increase absolute immunisation coverage by one percent was US$2.30.

Generally, community engagement interventions were successful in improving immunisation outcomes. The effects were robust to exclusion of high risk of bias studies. The effects were also uniform across geographies and baseline immunisation rates. Those who are designing and implementing interventions may wish to consider the following:

• Appropriate intervention design, including building in community engagement, can lead to intervention success. Notably, the dominant reasons for project success were positive intervention features and not external factors. Positive intervention features included local, supportive supervision, incentives for healthcare workers or caregivers, and health system integration and organisation. Many studies cited the effectiveness of the engagement strategy as a reason for the project success: Dialogues developed into action plans (Andersson et al., 2009) and needs assessments resulted in practical adjustments (Modi et al., 2019). Policymakers and implementers can attempt to integrate these positive intervention features into their projects. Methods for achieving sufficient intervention exposure should be integrated into the design of interventions; some interventions may require a longer implementation period to build community trust and buy-in, which may be essential for its effectiveness to be visible.
• Addressing common contextual barriers of immunisation and leveraging facilitators may be useful in designing new interventions. Ongoing monitoring or understanding of context can help in risk mitigation by making necessary modifications to the intervention design or implementation mid-way. For instance, the sensitisation of heads of households may address constraints imposed by social norms about decision making in the utilisation of healthcare services (Oche et al., 2011). Interventions that leverage facilitators may be more successful as a result of the presence of these facilitators but are threatened if the facilitators prove to be absent. For example, high maternal education may be leveraged in communication activities through the distribution of written materials. But the distribution of written literature in areas where maternal education is low will be unsuccessful.
• Implementation challenges could be avoided through appropriate intervention design. Policymakers and implementers should conduct scoping work to ensure that the interventions are practical to implement. Many interventions failed to account for existing implementation constraints and practicalities on the ground, such as limited cellphone service and insufficient staffing levels. This resulted in serious interruptions to implementation and low implementation fidelity. Administrative challenges, such as delayed approvals from local authorities, can be overcome or avoided through designs that include close collaboration with local partners.

In terms of implications for research, there are opportunities to conduct these studies with more rigorous evaluation designs, as asserted here. For instance, there are concerns related to reporting: In particular, for quasi-experimental studies, there is a lack of transparency with regard to how researchers addressed potential contamination or how they responded to implementation problems (e.g., attrition). Researchers may wish to attend to the following when undertaking IE in this area:

• Better reporting of interventions: For example, most studies lacked or had inadequate description and discussion of the intervention theory of change. This weakness limits the learning that can take place from the studies. Researchers should consider drawing on tools such as The Template for Intervention Description and Replication (TIDieR) reporting guidelines for health.
• Consideration of equity: There is a lack of research on how community engagement interventions affect immunisation outcomes by gender or income or for hard-to-reach populations. Sub-group analysis needs to be incorporated at the beginning of the evaluation and not as an afterthought. More disaggregated data by characteristics of interest, such as sex, socio-economics status, and religion, are needed.
• Prioritisation of mixed-methods IE and greater focus on intermediate outcomes: Drawing from qualitative work in mixed-methods studies, evaluators should be sure to report on why they think their interventions worked, not just if the interventions worked. Moreover, greater focus on effect of interventions on intermediate outcomes, in addition to final immunisation outcomes, will improve the learnings on mechanisms of change.
• Improved and standardised reporting of cost data and analysis: From the perspective of these researchers, reporting of total costs, average costs, and marginal costs per impact, and the cost per vaccine dose per child, all should be reported for evaluated immunisation programmes. There is a wide range of available guidance to assist researchers in this reporting.
• Donors' role: Research teams are more likely to plan and report on sub-group and cost analysis when the when the donor requires it.

Editor's note: On June 30 2022, an edition of the International Initiative for Impact Evaluation (3ie) Evidence Dialogues featured a brief presentation of the review's findings, along with a panel discussion highlighting how policymakers and health sector specialists can translate these approaches into practice. Panellists include: Tove Ryman, Senior Program Officer, Bill & Melinda Gates Foundation; Monica Jain, Lead Evaluation Specialist, 3ie; Lisa Menning, Department of Immunization, Vaccines and Biologicals, WHO; and Sebastian Martinez, Director, Evaluation, 3ie. Click on the video below to watch the hour-long recording.