How to Measure the Impact of Poverty Alleviation: In Search of New Methods in Evaluation

Written by Henri Jorritsma, Deputy Director of the IOB Policy and Operations Evaluation Department of the Dutch Ministry of Foreign Affairs, this editorial reviews the methodological challenges facing evaluation in the context of development aid's increasing focus on poverty. Jorritsma argues that, in part as a response to the Millennium Development Goals (MDGs)' call to reduce the proportion of people living on less than US$1 a day to half the 1990 level by 2015, development efforts are increasingly being directed at reducing poverty.

The challenge associated with this call, however, is as follows: "There is a general understanding that poverty is a complex multidimensional concept and that poverty reduction strategies should be based on a thorough analysis of its root causes. In other words, progress on such Goals as basic education or primary health care can, of course, be evaluated, but that does not automatically give much insight into the improvements of the poverty situation as such."

What are the implications of this challenge for evaluation? Jorritsma highlights a response by describing the experience of the IOB Policy and Operations Evaluation Deparment, which helps evaluate the Ministry of Foreign Affairs' efforts to improve the living conditions of poor groups worldwide. According to Jorritsma, emphasis in evaluating these projects has been placed on the degree to which the project goals have been achieved (measuring effectiveness) and project or programme execution (measuring efficiency of implementation). Less attention is paid to "processes that led to this situation and the underlying causes of poverty; the perception of the population regarding the poverty situation, its underlying causes and the relevance of aid activities; and the extent to which aid activities aimed at removing the causes of poverty as perceived by the population....Due to time constraints, discussions with target groups remain largely anecdotal."

Evaluation authorities, Jorritsma claims, need to shift their focus away from performance evaluation of projects and programmes and toward impact. In this context, evaluators are faced with three methodological challenges:

1. how to go beyond the outcome level in evaluation and measure the impact on poverty level
2. how to find out what is actually going on at the grassroots in terms of micro- and macro-linkages
3. how to systematically integrate the perceptions of the group being addressed by the project in evaluations.

Jorritsma provides a few examples of aid organisations that have attempted to grapple with these challenges. In the participatory research initiative called Voices of the Poor, the World Bank collected the voices of more than 60,000 poor women and men from 60 countries with the aim of trying to understand poverty from the perspective of the poor themselves. Among the IOB's efforts are a workshop (in the planning phases as of this writing) that will focus directly on these challenges. Some of the questions that will be highlighted include: "Should poverty be defined in a holistic way, narrowed down to target group perceptions or predefined and/or restricted by the evaluator? What methods should be applied for perception studies: PRA [participatory rural appraisal], the Method for Impact Assessment of Poverty Alleviation Projects (MAPP), life history, transect, or something else? How do we measure impact of interventions on poverty without sufficient baseline data? Can this be done with counterfactual analysis or before-after comparison?"

Jorritsma concludes that "Only when these challenges have been addressed will this and other evaluations be able to go beyond the outcome level and measure the impact on poverty."