Prognostic predictions in psychosis: exploring the complementary role of machine learning models

Background Predicting outcomes in schizophrenia spectrum disorders is challenging due to the variability of individual trajectories. While machine learning (ML) shows promise in outcome prediction, it has not yet been integrated into clinical practice. Understanding how ML models (MLMs) can complement psychiatrists’ predictions and bridge the gap between MLM capabilities and practical use is key. Objective This vignette study aims to compare the performance of psychiatrists and MLMs in predicting short-term symptomatic and functional remission in patients with first-episode psychosis and explore whether MLMs can improve psychiatrists’ prognostic accuracy. Method 24 psychiatrists predicted symptomatic and functional remission probabilities at 10 weeks based on written baseline information from 66 patients in the OPtimization of Treatment and Management of Schizophrenia in Europe (OPTiMiSE) trial. ML-generated predictions based on these vignettes were then shared with psychiatrists, allowing them to adjust their estimates. Findings The predictive accuracy of the MLM was low but comparable to that of psychiatrists for symptomatic remission (MLM: 0.50, psychiatrists: 0.52) and comparable to that of psychiatrists for functional remission (MLM: 0.72, psychiatrists: 0.79). Inter-rater agreement was low but comparable for psychiatrists and the MLM. Although the MLM did not improve overall predictive accuracy, it showed potential in aiding psychiatrists with difficult-to-predict cases. However, psychiatrists struggled to recognise when to rely on the model’s output, and we were unable to determine a clear pattern in these cases based on their characteristics. Conclusions MLMs may have the potential to support psychiatric decision-making, particularly in difficult-to-predict cases, but at present, their effectiveness remains limited due to constraints in predictive accuracy and the ability to identify when to rely on the model’s output. Addressing these issues is crucial to improve the utility of MLMs and foster their integration into clinical practice. Clinical implications MLMs are best suited as supplementary tools, providing a second opinion while psychiatrists retain decision-making autonomy. Integrating predictions from both sources may help reduce individual biases and improve accuracy. This approach leverages the strengths of MLMs without compromising clinical responsibility.