Propensity Score Matching (PSM): An HC3 Research Primer

"Propensity Score Matching (PSM) is a statistical technique that allows researchers to more accurately measure SBCC [social and behaviour change communication] program impact and to make a strong case for causal attribution. Using the PSM approach increases the accuracy of impact measurement because it controls for unaccounted factors that might bias a person to respond favorably to a communication program."

This Health Communication Capacity (HC3) Research Primer describes the use of PSM for measuring SBCC impact, when it should be used, and what implementers should know. According to the document, sometimes a programme is too large and widespread to be able to use randomised control trial designs (RCTs) because there is no comparable group that was never exposed to elements of the communication programme, or a programme may reach mostly a group that might be predisposed in one or another to react a certain way (e.g., educated in a certain way or of a predominant gender or age). "PSM provides a way to take those differences and predispositions - individuals’ propensity for exposure - into consideration and to control for them when calculating program impact."

Through multiple regression analysis, characteristics most strongly related to programme exposure can be identified. As described in the primer, through surveying the data, "PSM then matches people in the survey sample who have the same characteristics that make them more or less likely to be exposed to the intervention. It then compares the extent of behavior change among similar people who were exposed (the treatment group) and those not exposed (the matched comparison group)....PSM gives us confidence that the only difference between the matched persons is the one we want to examine: exposure to a specific SBCC intervention. This allows researchers to evaluate behavior change while controlling for the variables that predispose some people to be exposed and to change. This way, without assigning some people to receive the program and denying it to others, researchers can be certain that the predisposing variables are not the reason that an individual responded positively to an SBCC program - rather, it was the program itself that had an effect on the individual’s behavior."

According to the primer, PSM can be used to control for people's response bias when a control group for a RCT is no possible. It can increase impact measurement accuracy by controlling for unaccounted factors that might bias responses favourably to a programme. "Propensity score matching approximates the conditions of a randomized control trial design by creating matched groups with statistically equal likelihood of exposure to an intervention. With this technique, researchers are able to create intervention and matched comparison groups where the only difference between them is exposure to the intervention."

Implementers should bear in mind that PSM can only account for variables observed and measured; it cannot account for hidden variables not identified in the survey.

The primer uses South Africa's Scrutinize campaign [See Related Summaries below] of 8 animated advertising segments (animerts) as an example: "Fifteen variables were used to create the propensity score index: age, sex, marital status, level of education, household wealth, poverty classification, employment status, frequency of television viewing, frequency of radio listening, frequency of reading newspapers, frequency of reading magazines, frequency of internet use, type of settlement (urban, peri-urban, tribal, farming), race and province. Of the 10.8 million sexually active South Africans aged 16-32 in the year of the survey, 32% (3.5 million people) could correctly recall the animert regarding multiple sexual partners. Based on this estimate, propensity score matching enabled the evaluators to estimate that 3.2% of that population, or over 111,000 people, avoided multiple sexual partners as a direct result of exposure to the Scrutinize campaign."