Abstract
Studies that generate real-world evidence on the effects of medical products through analysis of digital data collected in clinical practice provide key insights for regulators, payers, and other healthcare decision-makers. Ensuring reproducibility of such findings is fundamental to effective evidence-based decision-making. We reproduce results for 150 studies published in peer-reviewed journals using the same healthcare databases as original investigators and evaluate the completeness of reporting for 250. Original and reproduction effect sizes were positively correlated (Pearson’s correlation = 0.85), a strong relationship with some room for improvement. The median and interquartile range for the relative magnitude of effect (e.g., hazard ratiooriginal/hazard ratioreproduction) is 1.0 [0.9, 1.1], range [0.3, 2.1]. While the majority of results are closely reproduced, a subset are not. The latter can be explained by incomplete reporting and updated data. Greater methodological transparency aligned with new guidance may further improve reproducibility and validity assessment, thus facilitating evidence-based decision-making. Study registration number: EUPAS19636.
| Original language | English |
|---|---|
| Article number | 5126 |
| Pages (from-to) | 1-11 |
| Journal | Nature Communications |
| Volume | 13 |
| Issue number | 1 |
| DOIs | |
| Publication status | Published - Dec 2022 |
Bibliographical note
Funding Information:S.V.W. received salary support from grants to Brigham and Women’s Hospital from Boehringer Ingelheim, Johnson & Johnson and Novartis Pharmaceuticals for unrelated work. She was supported by NHLBI RO1HL141505 and NIA R01AG053302 during the conduct of this work. E.M.G., L.R.G., and M.G. are employees of and have stock options in Aetion, Inc. D.L.I. and E.P. were former employees of Aetion, Inc. during the implementation of this study. Dr. Patorno was supported by the National Institute on Aging (K08AG055670). She is investigator of an investigator-initiated grant to the Brigham and Women’s Hospital from Boehringer Ingelheim, not related to the topic of the submitted work. J.J.G. received salary support from grants from Eli Lilly and Company and Novartis Pharmaceuticals Corporation to the Brigham and Women’s Hospital and was a consultant to Optum, Inc., all for unrelated work during the conduct of this study. He has since become an employee of Johnson & Johnson. Brian Nosek is Executive Director of the Center for Open Science, a nonprofit technology and culture change organization with a mission to increase openness, integrity, and reproducibility of research. A.B. in an employee of GSK and was at study initiation was an employee of Pfizer. He is a shareholder and hold stock options in GSK and previously held stock and stock options at Pfizer. D.B. is an employee of UCB Pharma. J.M.F. was supported by NHLBI RO1HL141505 during the conduct of this study. She has since become an employee of Optum Epidemiology. M.P.L. was supported by NHLBI F32 HL149256. J.R.R. was a paid consultant to Aetion during the first year of this study for unrelated work. K.B.F. is supported by a senior salary support award from the Fonds de recherche du Québec – santé (Quebec Foundation for Research–Health) and a William Dawson Scholar award from McGill University. David Martin has since become an employee of Moderna. All other authors have no conflict of interest to declare.
Funding Information:
This project was funded by the non-profit Arnold Ventures, with support from Brigham and Women’s Hospital (SVW, SS). Drs. Wang and Schneeweiss were supported by NHLBI RO1HL141505 and NIA R01AG053302 during the conduct of this work.
Publisher Copyright:
© 2022, The Author(s).
Funding
S.V.W. received salary support from grants to Brigham and Women’s Hospital from Boehringer Ingelheim, Johnson & Johnson and Novartis Pharmaceuticals for unrelated work. She was supported by NHLBI RO1HL141505 and NIA R01AG053302 during the conduct of this work. E.M.G., L.R.G., and M.G. are employees of and have stock options in Aetion, Inc. D.L.I. and E.P. were former employees of Aetion, Inc. during the implementation of this study. Dr. Patorno was supported by the National Institute on Aging (K08AG055670). She is investigator of an investigator-initiated grant to the Brigham and Women’s Hospital from Boehringer Ingelheim, not related to the topic of the submitted work. J.J.G. received salary support from grants from Eli Lilly and Company and Novartis Pharmaceuticals Corporation to the Brigham and Women’s Hospital and was a consultant to Optum, Inc., all for unrelated work during the conduct of this study. He has since become an employee of Johnson & Johnson. Brian Nosek is Executive Director of the Center for Open Science, a nonprofit technology and culture change organization with a mission to increase openness, integrity, and reproducibility of research. A.B. in an employee of GSK and was at study initiation was an employee of Pfizer. He is a shareholder and hold stock options in GSK and previously held stock and stock options at Pfizer. D.B. is an employee of UCB Pharma. J.M.F. was supported by NHLBI RO1HL141505 during the conduct of this study. She has since become an employee of Optum Epidemiology. M.P.L. was supported by NHLBI F32 HL149256. J.R.R. was a paid consultant to Aetion during the first year of this study for unrelated work. K.B.F. is supported by a senior salary support award from the Fonds de recherche du Québec – santé (Quebec Foundation for Research–Health) and a William Dawson Scholar award from McGill University. David Martin has since become an employee of Moderna. All other authors have no conflict of interest to declare. This project was funded by the non-profit Arnold Ventures, with support from Brigham and Women’s Hospital (SVW, SS). Drs. Wang and Schneeweiss were supported by NHLBI RO1HL141505 and NIA R01AG053302 during the conduct of this work.