Detecting and explaining missing comparability in cross-national studies: The case of citizen evaluation of patriotism

Measurement invariance tests are an important precondition to analyze cross-national data. However, the traditional approach of multigroup confirmatory factor analysis (MGCFA) has been criticized as too strict and more liberal approaches, such as alignment, have been proposed. However, both approaches can only detect but cannot explain why there are comparability issues. Mixed methods approaches combining quantitative and qualitative insights from web probing provide a powerful tool to detect and explain a lack of comparability of measures. For this study, we selected the 2013 International Social Survey Program item battery on “Citizen Evaluation Of Patriotism” and assessed the comparability for Germany (N=1,717), Great Britain (N=904), the U.S. (N=1,274), Mexico (N=1,062), and Spain (N=1,225) and combined it with web probing results from an online survey conducted in 2014 in the five countries (N=2,685). Strict measurement invariance tests using MGCFA failed to show scalar measurement invariance but with an approximate approach of alignment estimation unbiased equal factor loadings and latent means could be estimated for all countries. In line with MGCFA results, qualitative web probing detected issues that question the comparability of results.