Undertaking Rapid Assessments and Real-time Monitoring in the COVID-19 Context: Lessons from UNICEF South Asia

"Few studies have focused on the impact of the pandemic on evidence generation, and explored the learning around the methods and approaches that can be adopted to generate rapid evidence in an emergency context, based on their practical application on the ground."

In an unprecedented situation such as the COVID-19 pandemic, information is required to quickly and repeatedly assess the situation on the ground. This assessment enables organisations such as the United Nations Children's Fund (UNICEF) to design solutions and advocate for actions to meet the needs of vulnerable populations, including women and children. In 2020, UNICEF Country Offices across South Asia conducted a variety of rapid assessments and real-time evidence-generating exercises. Drawing on the documented experiences of nine rapid assessments covering six countries (Afghanistan, Bangladesh, India, Nepal, Pakistan, and Sri Lanka), this report by the Evaluation Unit of the UNICEF Regional Office for South Asia (ROSA) and the Research and Evaluation team at UNICEF India shares insights with regard to the design and implementation of rapid assessments in a pandemic context. The lessons and recommendations presented in this report may be of interest to monitoring and evaluation (M&E) specialists, programme planners, and other technical and non-technical audiences to inform future evidence-generation activities in emergencies.

After a presentation of the background and context of the report (Chapter 1), Chapter 2 provides an overview of the nine documented rapid assessments and the methodology adopted for the selection of cases, data collection, analysis, synthesis, and documentation. In short, the case studies (amended in full as the second portion of the main report) focus on these topics:

1. Assessing the immediate impact of COVID-19 among the most vulnerable in Uttar Pradesh, India
2. Child and family tracker: A case study in Nepal
3. Community-based monitoring (CBM) to assess the socio-economic impact of the COVID-19 pandemic on vulnerable populations in India
4. COVID-19-related risk communication and community engagement (RCCE) behavioural change study in Pakistan
5. Impact of the COVID-19 crisis on households in Sri Lanka
6. Insights and feedback on Coronavirus RCCE in Bangladesh
7. Rapid assessment of learning during school closures across six states of India in the context of COVID-19
8. Rapid assessment of the socio-economic impact of COVID-19 in Herat Province, Afghanistan
9. Understanding youth perceptions of COVID-19 in Pakistan

As lockdowns were in place, and the risk of transmission of the virus needed to be mitigated, traditional face-to-face approaches for data collection could not be adopted. Thus, in most cases, remote data collection via telephone surveys, interactive voice response (IVR), and online surveys was used. All the assessments used quantitative methods; some used mixed-methods for additional qualitative data collection. Most assessments directly surveyed household members or sampled individuals; three studies also used community members as "informants". A range of sampling approaches were applied and combined, including grid-based, stratified random, purposive, and convenience sampling strategies.

Chapter 3 presents a synthesis of the strengths, limitations, trade-offs, innovations, and lessons learned in designing and implementing the nine studies; cross-cutting issues of gender and equity are also discussed.

• Section 3.1 presents findings on data collection and analysis - selected lessons:
  • Data collection modalities need to be selected based on the scope of the survey, profile of the respondents, required response rate, timelines, budget, opportunities (e.g., access to phone numbers, access to technology) and context.
  • Questions need to be prioritised based on the objectives of the assessment and intended use of the data. Phone surveys are to be favoured over IVR/online surveys when questionnaires are longer, but they are subject to survey fatigue.
  • Different data collection modalities and approaches require different levels and types of capacities to implement the survey. A community-based approach using a network of civil society organisations (CSOs) and community volunteers (CVs) with local presence can effectively gather data among vulnerable populations but requires considerable investment in training and follow-up, as well as in-house capacity.
  • Generating data from multiple sources requires the necessary capacity to process, analyse, and report the data in a mixed/triangulated way. When including a limited number of qualitative interviews as part of rapid assessments, it is important to carefully select informants who are knowledgeable and can provide insights about the situation on the ground.
  • In the case of longitudinal assessments, debriefing sessions should be scheduled after each data collection round.
  • Pre-testing the questionnaire among survey respondents should be factored into the study design.
  • Only information that is relevant to the study objectives should be gathered.
• Section 3.2 covers sampling approaches, sharing lessons learned such as: Remote data collection has an inherent limitation in reaching the most vulnerable, who are less likely to have mobile phone/internet access. Nonetheless, sampling strategies can improve their representation through, for example, purposeful sampling, use of quotas, and making sample frames more inclusive. However, this comes with trade-offs of reducing the randomness of the sample (and therefore introducing bias) and additional investment in time and resources.
• Section 3.3 discusses partnerships and collaborations for designing and implementing the studies, sharing lessons learned such as: The technical expertise, networks, and resources of local partners can be leveraged for remote data collection that, in the context of a pandemic, can help roll out a remote survey, meet the short timelines, promote inclusiveness, and contribute to the robustness of the findings.
• Section 3.4 focuses on agility and timeliness of the studies, sharing lessons learned such as: To ensure that the process is more agile, particularly for contracting external research and data collection capacity, a long-term agreement (LTA) with technical partners should be in place.
• Section 3.5 covers the dissemination and use of findings, sharing lessons learned such as: There is a need to identify potential partners/users and engage with the intended users from the start of the study, gather data with attention to use, present evidence at the level of different types of audiences (technical and non-technical), and share the evidence in a timely way. Survey findings can be shared with a range of stakeholders during formal and informal events, through a range of materials (e.g., advocacy briefs, fact sheets, and policy briefs), and more widely on websites and media (e.g., newspapers and radio), as well as with respondents in the community during the survey.

Drawing on the cases documented in this synthesis report, Chapter 4 presents lessons and recommendations for planning, designing, and implementing rapid assessments in a pandemic situation, and the dissemination and use of the evidence in such contexts. For example: The principle of "do no harm" in data collection should guide planning, design, and implementation of the rapid assessment. It is important to have the initial proposal or concept note reviewed by someone with expertise in the ethical dimensions of evidence generation. When required (e.g., surveys covering children and adolescents), a review by an ethics review board (ERB) should be included in the workplan.

Overall, the cases provide good examples of the use of findings. For example, the insights and recommendations around COVID-19 in the RCCE-Pakistan case were used by the Pakistan's National COVID-19 RCCE Taskforce during discussions on the COVID-19 response and for external communication.

A positive consequence of the collaborations described in this report has been the opportunity to build the capacity of local partners. In the RCCE-Pakistan case, for example, the partner took on board the weight model developed by UNICEF headquarters (HQ) to apply it in subsequent survey rounds, while in the CBM-India case, UNICEF built the capacity of the CSOs and anchors, creating a cadre of local volunteers who are now trained and sensitive to the need for quality data and can be leveraged to monitor future interventions.

Note: As part of the 2021 gLOCAL Evaluation week (May 31-June 4 2021), UNICEF South Asia and Country Offices presented experiences and lessons learned from implementing rapid assessments and real-time monitoring approaches, featuring the nine country cases referenced above. Click on the video below to view the webinar.

Click here and use the dropdown menu on the right to download the main report, summary brief, and country-specific case studies.