An open source machine learning framework for efficient and transparent systematic reviews

Rens van de Schoot; Jonathan de Bruin; Raoul Schram; Parisa Zahedi; Jan de Boer; Felix Weijdema; Bianca Kramer; Martijn Huijts; Maarten Hoogerwerf; Gerbrich Ferdinands; Albert Harkema; Joukje Willemsen; Yongchao Ma; Qixiang Fang; Sybren Hindriks; Lars Tummers; Daniel Oberski

doi:10.1038/s42256-020-00287-7

An open source machine learning framework for efficient and transparent systematic reviews

Rens van de Schoot
, Jonathan de Bruin
, Raoul Schram
, Parisa Zahedi
, Jan de Boer
, Felix Weijdema
, Bianca Kramer
, Martijn Huijts
, Maarten Hoogerwerf
, Gerbrich Ferdinands
, Albert Harkema
, Joukje Willemsen
, Yongchao Ma
, Qixiang Fang
, Sybren Hindriks
, Lars Tummers
, Daniel Oberski

Research output: Contribution to journal › Article › Academic › peer-review

Abstract

To help researchers conduct a systematic review or meta-analysis as efficiently and transparently as possible, we designed a tool to accelerate the step of screening titles and abstracts. For many tasks—including but not limited to systematic reviews and meta-analyses—the scientific literature needs to be checked systematically. Scholars and practitioners currently screen thousands of studies by hand to determine which studies to include in their review or meta-analysis. This is error prone and inefficient because of extremely imbalanced data: only a fraction of the screened studies is relevant. The future of systematic reviewing will be an interaction with machine learning algorithms to deal with the enormous increase of available text. We therefore developed an open source machine learning-aided pipeline applying active learning: ASReview. We demonstrate by means of simulation studies that active learning can yield far more efficient reviewing than manual reviewing while providing high quality. Furthermore, we describe the options of the free and open source research software and present the results from user experience tests. We invite the community to contribute to open source projects such as our own that provide measurable and reproducible improvements over current practice.

Original language	English
Pages (from-to)	125-133
Number of pages	9
Journal	Nature Machine Intelligence
Volume	3
Issue number	2
DOIs	https://doi.org/10.1038/s42256-020-00287-7
Publication status	Published - Feb 2021

Bibliographical note

Funding Information:
We would like to thank the Utrecht University Library, focus area Applied Data Science, and departments of Information and Technology Services, Test and Quality Services, and Methodology and Statistics, for their support. We also want to thank all researchers who shared data, participated in our user experience tests or who gave us feedback on ASReview in other ways. Furthermore, we would like to thank the editors and reviewers for providing constructive feedback. This project was funded by the Innovation Fund for IT in Research Projects, Utrecht University, the Netherlands.

Publisher Copyright:
© 2021, The Author(s).

Copyright:
Copyright 2021 Elsevier B.V., All rights reserved.

Funding

We would like to thank the Utrecht University Library, focus area Applied Data Science, and departments of Information and Technology Services, Test and Quality Services, and Methodology and Statistics, for their support. We also want to thank all researchers who shared data, participated in our user experience tests or who gave us feedback on ASReview in other ways. Furthermore, we would like to thank the editors and reviewers for providing constructive feedback. This project was funded by the Innovation Fund for IT in Research Projects, Utrecht University, the Netherlands.

Keywords

Active learning
Machine learning
Open science
Researcher-in-the-loop
Systematic reviewing

Access to Document

10.1038/s42256-020-00287-7Licence: CC BY

s42256-020-00287-7Final published version, 1.22 MBLicence: CC BY

Cite this

Schoot, R. V. D., Bruin, J. D., Schram, R., Zahedi, P., Boer, J. D., Weijdema, F., Kramer, B., Huijts, M., Hoogerwerf, M., Ferdinands, G., Harkema, A., Willemsen, J., Ma, Y., Fang, Q., Hindriks, S., Tummers, L., & Oberski, D. (2021). An open source machine learning framework for efficient and transparent systematic reviews. Nature Machine Intelligence, 3(2), 125-133. https://doi.org/10.1038/s42256-020-00287-7

@article{c0a4dbb03630427b8405ff397dfd50d8,

title = "An open source machine learning framework for efficient and transparent systematic reviews",

abstract = "To help researchers conduct a systematic review or meta-analysis as efficiently and transparently as possible, we designed a tool to accelerate the step of screening titles and abstracts. For many tasks—including but not limited to systematic reviews and meta-analyses—the scientific literature needs to be checked systematically. Scholars and practitioners currently screen thousands of studies by hand to determine which studies to include in their review or meta-analysis. This is error prone and inefficient because of extremely imbalanced data: only a fraction of the screened studies is relevant. The future of systematic reviewing will be an interaction with machine learning algorithms to deal with the enormous increase of available text. We therefore developed an open source machine learning-aided pipeline applying active learning: ASReview. We demonstrate by means of simulation studies that active learning can yield far more efficient reviewing than manual reviewing while providing high quality. Furthermore, we describe the options of the free and open source research software and present the results from user experience tests. We invite the community to contribute to open source projects such as our own that provide measurable and reproducible improvements over current practice.",

keywords = "Active learning, Machine learning, Open science, Researcher-in-the-loop, Systematic reviewing",

author = "Schoot, \{Rens van de\} and Bruin, \{Jonathan de\} and Raoul Schram and Parisa Zahedi and Boer, \{Jan de\} and Felix Weijdema and Bianca Kramer and Martijn Huijts and Maarten Hoogerwerf and Gerbrich Ferdinands and Albert Harkema and Joukje Willemsen and Yongchao Ma and Qixiang Fang and Sybren Hindriks and Lars Tummers and Daniel Oberski",

note = "Funding Information: We would like to thank the Utrecht University Library, focus area Applied Data Science, and departments of Information and Technology Services, Test and Quality Services, and Methodology and Statistics, for their support. We also want to thank all researchers who shared data, participated in our user experience tests or who gave us feedback on ASReview in other ways. Furthermore, we would like to thank the editors and reviewers for providing constructive feedback. This project was funded by the Innovation Fund for IT in Research Projects, Utrecht University, the Netherlands. Publisher Copyright: {\textcopyright} 2021, The Author(s). Copyright: Copyright 2021 Elsevier B.V., All rights reserved.",

year = "2021",

month = feb,

doi = "10.1038/s42256-020-00287-7",

language = "English",

volume = "3",

pages = "125--133",

journal = "Nature Machine Intelligence",

issn = "2522-5839",

publisher = "Springer International Publishing",

number = "2",

}

Schoot, RVD, Bruin, JD, Schram, R, Zahedi, P, Boer, JD, Weijdema, F, Kramer, B, Huijts, M, Hoogerwerf, M, Ferdinands, G, Harkema, A, Willemsen, J, Ma, Y, Fang, Q, Hindriks, S, Tummers, L & Oberski, D 2021, 'An open source machine learning framework for efficient and transparent systematic reviews', Nature Machine Intelligence, vol. 3, no. 2, pp. 125-133. https://doi.org/10.1038/s42256-020-00287-7

TY - JOUR

T1 - An open source machine learning framework for efficient and transparent systematic reviews

AU - Schoot, Rens van de

AU - Bruin, Jonathan de

AU - Schram, Raoul

AU - Zahedi, Parisa

AU - Boer, Jan de

AU - Weijdema, Felix

AU - Kramer, Bianca

AU - Huijts, Martijn

AU - Hoogerwerf, Maarten

AU - Ferdinands, Gerbrich

AU - Harkema, Albert

AU - Willemsen, Joukje

AU - Ma, Yongchao

AU - Fang, Qixiang

AU - Hindriks, Sybren

AU - Tummers, Lars

AU - Oberski, Daniel

N1 - Funding Information: We would like to thank the Utrecht University Library, focus area Applied Data Science, and departments of Information and Technology Services, Test and Quality Services, and Methodology and Statistics, for their support. We also want to thank all researchers who shared data, participated in our user experience tests or who gave us feedback on ASReview in other ways. Furthermore, we would like to thank the editors and reviewers for providing constructive feedback. This project was funded by the Innovation Fund for IT in Research Projects, Utrecht University, the Netherlands. Publisher Copyright: © 2021, The Author(s). Copyright: Copyright 2021 Elsevier B.V., All rights reserved.

PY - 2021/2

Y1 - 2021/2

N2 - To help researchers conduct a systematic review or meta-analysis as efficiently and transparently as possible, we designed a tool to accelerate the step of screening titles and abstracts. For many tasks—including but not limited to systematic reviews and meta-analyses—the scientific literature needs to be checked systematically. Scholars and practitioners currently screen thousands of studies by hand to determine which studies to include in their review or meta-analysis. This is error prone and inefficient because of extremely imbalanced data: only a fraction of the screened studies is relevant. The future of systematic reviewing will be an interaction with machine learning algorithms to deal with the enormous increase of available text. We therefore developed an open source machine learning-aided pipeline applying active learning: ASReview. We demonstrate by means of simulation studies that active learning can yield far more efficient reviewing than manual reviewing while providing high quality. Furthermore, we describe the options of the free and open source research software and present the results from user experience tests. We invite the community to contribute to open source projects such as our own that provide measurable and reproducible improvements over current practice.

AB - To help researchers conduct a systematic review or meta-analysis as efficiently and transparently as possible, we designed a tool to accelerate the step of screening titles and abstracts. For many tasks—including but not limited to systematic reviews and meta-analyses—the scientific literature needs to be checked systematically. Scholars and practitioners currently screen thousands of studies by hand to determine which studies to include in their review or meta-analysis. This is error prone and inefficient because of extremely imbalanced data: only a fraction of the screened studies is relevant. The future of systematic reviewing will be an interaction with machine learning algorithms to deal with the enormous increase of available text. We therefore developed an open source machine learning-aided pipeline applying active learning: ASReview. We demonstrate by means of simulation studies that active learning can yield far more efficient reviewing than manual reviewing while providing high quality. Furthermore, we describe the options of the free and open source research software and present the results from user experience tests. We invite the community to contribute to open source projects such as our own that provide measurable and reproducible improvements over current practice.

KW - Active learning

KW - Machine learning

KW - Open science

KW - Researcher-in-the-loop

KW - Systematic reviewing

UR - https://www.scopus.com/pages/publications/85100105574

U2 - 10.1038/s42256-020-00287-7

DO - 10.1038/s42256-020-00287-7

M3 - Article

SN - 2522-5839

VL - 3

SP - 125

EP - 133

JO - Nature Machine Intelligence

JF - Nature Machine Intelligence

IS - 2

ER -

An open source machine learning framework for efficient and transparent systematic reviews

Abstract

Bibliographical note

Funding

Keywords

Access to Document

Other files and links

Fingerprint

Artificial intelligence supports literature screening in medical guideline development: towards up-to-date medical guidelines

ASReview CNN HPO plugin

ASReview model switcher plugin

ASReview wide doc2vec plugin

Code repository for: "Evaluation of the performance of neural network models and classical models in the context of Active Learning for Systematic Reviewing"

Cite this