A permutation test to probe the statistical significance of sudden gain frequency: An application to patterns of change in cognitive and interpersonal therapy for depression

Objectives: Sudden gains were first identified in cognitive therapy (CT) for depression to characterize large and stable improvements. They have subsequently been studied in at least 50 studies across a range of disorders and treatments. There is evidence that sudden gains are reliably associated with positive treatment outcomes. Nonetheless, simulations have suggested that sudden gains may occur spuriously. We propose the use of a permutation test to yield a p-value to ascertain whether sudden gains occur above and beyond the level that can be expected by chance in a given sample.

Methods: We reanalyzed the study by Lemmens et al. (2016) to explore the utility of a permutation test for the occurrence and effects of sudden gains. We permuted session-by-session depression scores within each session before identifying sudden gains to explore the effects of the number of identified sudden gains. We also permuted sudden gains status. For both analyses we resampled 10,000 random datasets.

Results: In the permuted samples, the mean number of identified sudden gains was 29.43, replicating prior work that suggests caution about the occurrence of sudden gains. However, this rate was lower than the number of gains (n = 52) occurring in the original data (p < 0.001). The association between sudden gains and outcomes was also above and beyond chance level (p < 0.001).

Discussion: These data suggest that sudden gains occur above and beyond chance level. We provide code for these analyses and updated the suddengains R package to facilitate replications and further method development.