Evaluating propensity score balance measures in typical pharmacoepidemiological settings

Background: Several propensity score (PS) balance measures have been compared in simulated data with normally distributed covariates. Comparisons in data with binary or mixed covariate distributions and rare outcomes, typical of pharmacoepidemiologic data sets, are scarce. Objectives: To compare balance measures in simulated data with various covariate distributions and rare outcomes. Methods: We performed Monte Carlo simulations to examine the relative ability of different balance measures to select PS models that yielded the least biased estimates. In different simulations, covariates were binary, normal or gamma distributed, considering sample sizes of n = 400, 1,600, and 3,000, incidence of outcomes of 10% and 25%, and strength of exposure-outcome association of OR = 1 and 2. Bias was estimated as the difference between the true marginal effect and the effect estimate obtained from a logistic regression model with PS as a covariate. The balance of covariates between treatment groups was assessed using the standardised difference (SD), Kolmogorov Smirnov (KS) distance, Lévy distance (Lévy) and overlapping coefficient (OVL). Pearson's correlation coefficients (r) between these balance measures and bias were calculated. Results: With large sample sizes, all balance measures were similarly correlated with bias irrespective of covariate distributions, strength of the effect, and prevalence of outcome (e.g., when all covariates were binary, OR = 2.0, n = 3,000 and incidence of outcome = 25%: correlations were 0.76, 0.79, 0.79, and -0.79 for SD, KSD, Lévy and OVL, respectively). These correlations decreased with smaller sample sizes (e.g., for n = 400: 0.51, 0.20, 0.17, and -0.43, for SD, KSD, Lévy distance and OVL, respectively). Incidence of the outcome and strength of the exposure-outcome relation didn't have much impact. For sample sizes smaller than 800, SD showed a better correlation with bias than the other balance measures. Conclusions: The SD or KS performed best across different simulation scenarios and are recommended for reporting the amount of balance reached and selecting the final PS model.