Back to Annual Meeting

Back to Annual Meeting

APHA Scientific Session and Event Listing

Ali Karim, PhD¹, Michael J. McQuestion, MS, MPH, PhD², and Timothy Williams, MA, MEM¹. (1) John Snow, Inc., 1616 North Fort Myer Drive, 11th Floor, Arlington, VA 22209, 7035287474, ali_karim@jsi.com, (2) Department of Population & Family Health Sciences, Johns Hopkins Bloomberg School of Public Health, 615 N. Wolfe Street, Baltimore, MD 21205

Designing an appropriate impact evaluation strategy in a post intervention situation is problematic, especially for full-coverage programs, or, when program placement is non-random. In such cases, program impact is often measured by observing the correlation of the expected outcome with the variability in program scope and intensity across the different segments of the target population; netting out the bias due to non-random program placement by using econometric models. However, in such cases, measuring program exposure remains a challenge. While the systematic error in measuring program exposure that is correlated with outcome can be purged by econometric models, the non-systematic (or random) error of measuring exposure haunts the validity of the evaluation.

To evaluate the impact of the $57 million Bill Gates funded African Youth Alliance (AYA), a program to improve adolescent reproductive health in four African countries, we design a posttest only study addressing the methodological concerns discussed above. We collect information from unmarried 17- to 22-year-old youths, households, and communities from Ghana, Tanzania, and Uganda, covering areas where AYA was implemented over the past three years in different scope and intensity. The sample size of the young adults for each country varied between 1,800 to 3,700, depending on the expected number of unmarried 17- to 22-year-old girls or boys per household. We measure program exposure at the community- as well at the individual-level and obtain its impact on sexual behavior and its antecedents by applying two different methods, instrumental variable regression and propensity score matching technique, to account for the bias associated with non-random program placement. Two different program exposure measures with two different analytic technique gives four sets of impact analyses for each of the outcomes considered. The analysis is presented by gender; and the implications of the assumptions associated with the four different methodologies on the conclusion of the findings are discussed using selected outcomes, as examples.

Learning Objectives: At the conclusion of the session, the participant (learner) in this session will be able to

• Identify different ways of measuring program exposure and their strengths and weaknesses
• Acknowledge the contemporary methods in evaluating program impact using posttest-only design, including their strengths and weaknesses
• Draw plausible conclusion on program impact by triangulated results from four different analyses, given their strengths and weaknesses

Keywords: Evaluation, Methodology

Related Web page: www.ayaonline.org/

Presenting author's disclosure statement:

Any relevant financial relationships? No