Measuring the impact of capacity development on health: Pact's organizational performance index

The impact of capacity development work is notoriously difficult to measure. Many capacity developers analyze the results of their work by measuring changes in organizational systems, practices, policies and staff skills. Capacity development practitioners must go further, connecting internal organizational change to improvements in the health of beneficiaries and communities. We developed an Organizational Performance Index (OPI) that is designed to capture an organization's performance based on a scale of Level 1 (low performing) to Level 4 (high performing) under four domains of performance: effectiveness, efficiency, relevance, and sustainability and tested its reliability. Methods: We evaluated the reliability of the OPI with programs in Ethiopia, Nigeria, South Sudan, Swaziland, Vietnam, and Zimbabwe. In total, 40 partners were assessed, with each partner assessed by two staff as well as self-assessed by the organization's own staff. Cronbach's Alpha was used to assess the inter-rater reliability of the scoring. Reliability of .70 or higher was sought in the research. Results: The reliability of the OPI tool was on average .85 for all countries with only Nigeria and Ethiopia reporting scores under .70. Discussion: The OPI proved reliable and can be utilized by capacity developers to independently evaluate capacity development outcomes of their health projects. Further, the OPI measured change in organizational performance and correlated capacity development inputs and community level impact for health projects. It is a revolutionary yet user-friendly approach that looks beyond the development of organizational systems or skills and analyzes the actual value added through capacity development efforts.