Evaluating Recommender Systems: Survey and Framework

Eva Zangerle*, Christine Bauer

*Corresponding author for this work

Research output: Contribution to journalArticleAcademicpeer-review


The comprehensive evaluation of the performance of a recommender system is a complex endeavor: many facets need to be considered in configuring an adequate and effective evaluation setting. Such facets include, for instance, defining the specific goals of the evaluation, choosing an evaluation method, underlying data, and suitable evaluation metrics. In this article, we consolidate and systematically organize this dispersed knowledge on recommender systems evaluation. We introduce the Framework for Evaluating Recommender systems (FEVR), which we derive from the discourse on recommender systems evaluation. In FEVR, we categorize the evaluation space of recommender systems evaluation. We postulate that the comprehensive evaluation of a recommender system frequently requires considering multiple facets and perspectives in the evaluation. The FEVR framework provides a structured foundation to adopt adequate evaluation configurations that encompass this required multi-facetedness and provides the basis to advance in the field. We outline and discuss the challenges of a comprehensive evaluation of recommender systems and provide an outlook on what we need to embrace and do to move forward as a research community.

Original languageEnglish
Article number170
Pages (from-to)1–38
Number of pages38
JournalACM Computing Surveys
Issue number8
Publication statusPublished - 23 Dec 2022


  • FEVR
  • Framework for EValuating Recommender systems
  • evaluation
  • recommender systems
  • survey


Dive into the research topics of 'Evaluating Recommender Systems: Survey and Framework'. Together they form a unique fingerprint.

Cite this