Strengthening causal inference using Optimal Full Matching: An Empirical Demonstration in a Program Evaluation

Program evaluations are often limited by confounding due to differences in treatment and control groups. Optimal full matching, a recent innovation in propensity score matching, can minimize these differences and strengthen causal inferences. However, optimal full matching makes computing the treatment effect and statistical significance challenging because matched sets, not pairs, are created. Parametric methods require a high degree of homogeneity of outcomes within matched sets. Using data from the New York/New York III supportive housing program evaluation we assessed the usefulness of optimal full matching, stratification, and one-to-one greedy matching in minimizing differences between people eligible for the program who were placed versus not placed in housing. For example, among 201 young adults who aged out of foster care, were eligible for the program (2007-2008), and had one year of follow-up time, 68 were placed and 133 were not placed. On average there was 65% of a standard deviation difference between baseline characteristics of treatment and control groups before matching. Optimal full matching produced the largest reduction, with groups having an average of 4% of a standard deviation difference post-matching, compared to stratification (13%) and one-to-one greedy matching (36%). In our dataset the cost data for the main outcomes were heavily skewed and weakly correlated within matched sets (the intraclass correlation coefficient for total cost of services and benefit post-treatment was 0.004). Nonparametric permutation-based methods (Hodges-Lehmann aligned rank test and bootstrapping) addressed this challenge. Optimal full matching can strengthen causal inferences when randomization is not feasible.