SYMBALS: A Systematic Review Methodology Blending Active Learning and Snowballing

Max van Haastrecht, Injy Sarhan, Bilge Yigit Ozkan, Matthieu Brinkhuis, Marco Spruit

Research output: Contribution to journalArticleAcademicpeer-review


Research output has grown significantly in recent years, often making it difficult to see the forest for the trees. Systematic reviews are the natural scientific tool to provide clarity in these situations. However, they are protracted processes that require expertise to execute. These are problematic characteristics in a constantly changing environment. To solve these challenges, we introduce an innovative systematic review methodology: SYMBALS. SYMBALS blends the traditional method of backward snowballing with the machine learning method of active learning. We applied our methodology in a case study, demonstrating its ability to swiftly yield broad research coverage. We proved the validity of our method using a replication study, where SYMBALS was shown to accelerate title and abstract screening by a factor of 6. Additionally, four benchmarking experiments demonstrated the ability of our methodology to outperform the state-of-the-art systematic review methodology FAST2.
Original languageEnglish
Article number685591
Pages (from-to)1-14
Number of pages14
JournalFrontiers in Research Metrics and Analytics
Publication statusPublished - 28 May 2021


  • systematic review
  • methodology
  • active learning
  • machine learning
  • backward snowballing


Dive into the research topics of 'SYMBALS: A Systematic Review Methodology Blending Active Learning and Snowballing'. Together they form a unique fingerprint.

Cite this