Towards a Comprehensive Human-Centred Evaluation Framework for Explainable AI

Ivania Donoso-Guzmán; Jeroen Ooge; Denis Parra; Katrien Verbert

doi:10.1007/978-3-031-44070-0_10

Towards a Comprehensive Human-Centred Evaluation Framework for Explainable AI

Ivania Donoso-Guzmán^*, Jeroen Ooge, Denis Parra, Katrien Verbert

^*Corresponding author for this work

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › Academic › peer-review

Abstract

While research on explainable AI (XAI) is booming and explanation techniques have proven promising in many application domains, standardised human-centred evaluation procedures are still missing. In addition, current evaluation procedures do not assess XAI methods holistically in the sense that they do not treat explanations’ effects on humans as a complex user experience. To tackle this challenge, we propose to adapt the User-Centric Evaluation Framework used in recommender systems: we integrate explanation aspects, summarise explanation properties, indicate relations between them, and categorise metrics that measure these properties. With this comprehensive evaluation framework, we hope to contribute to the human-centred standardisation of XAI evaluation.

Original language	English
Title of host publication	Explainable Artificial Intelligence - 1st World Conference, xAI 2023, Proceedings
Editors	Luca Longo
Publisher	Springer
Pages	183-204
Number of pages	22
ISBN (Print)	9783031440694
DOIs	https://doi.org/10.1007/978-3-031-44070-0_10
Publication status	Published - 21 Oct 2023
Externally published	Yes
Event	1st World Conference on eXplainable Artificial Intelligence, xAI 2023 - Lisbon, Portugal Duration: 26 Jul 2023 → 28 Jul 2023

Publication series

Name	Communications in Computer and Information Science
Volume	1903 CCIS
ISSN (Print)	1865-0929
ISSN (Electronic)	1865-0937

Conference

Conference	1st World Conference on eXplainable Artificial Intelligence, xAI 2023
Country/Territory	Portugal
City	Lisbon
Period	26/07/23 → 28/07/23

Bibliographical note

Publisher Copyright:
© 2023, The Author(s), under exclusive license to Springer Nature Switzerland AG.

Funding

This work was partially funded by ANID Chile, Millennium Science Initiative Program, codes ICN2021 004 (iHealth) and ICN17 002 (IMFD), by Basal Funds for Center of Excellence FB210017 (CENIA), the Research Foundation Flanders (FWO, grant G0A3319N) and KU Leuven (grant C14/21/072). In addition, we thank Fondecyt grant 1231724. The research of Ivania Donoso-Guzmán was supported by the doctoral scholarship of ANID Chile.

Funders	Funder number
Basal Funds for Center of Excellence	FB210017
CENIA
IMFD
Fondo Nacional de Desarrollo Científico y Tecnológico	1231724
Fonds Wetenschappelijk Onderzoek	G0A3319N
KU Leuven	C14/21/072
Agencia Nacional de Investigación y Desarrollo	ICN17 002, ICN2021 004

Keywords

Evaluation framework
Human-centred evaluation
XAI Evaluation

Access to Document

10.1007/978-3-031-44070-0_10

Cite this

Donoso-Guzmán, I., Ooge, J., Parra, D., & Verbert, K. (2023). Towards a Comprehensive Human-Centred Evaluation Framework for Explainable AI. In L. Longo (Ed.), Explainable Artificial Intelligence - 1st World Conference, xAI 2023, Proceedings (pp. 183-204). (Communications in Computer and Information Science; Vol. 1903 CCIS). Springer. https://doi.org/10.1007/978-3-031-44070-0_10

@inproceedings{9ca1be1dd9394560902658b78ffa673a,

title = "Towards a Comprehensive Human-Centred Evaluation Framework for Explainable AI",

abstract = "While research on explainable AI (XAI) is booming and explanation techniques have proven promising in many application domains, standardised human-centred evaluation procedures are still missing. In addition, current evaluation procedures do not assess XAI methods holistically in the sense that they do not treat explanations{\textquoteright} effects on humans as a complex user experience. To tackle this challenge, we propose to adapt the User-Centric Evaluation Framework used in recommender systems: we integrate explanation aspects, summarise explanation properties, indicate relations between them, and categorise metrics that measure these properties. With this comprehensive evaluation framework, we hope to contribute to the human-centred standardisation of XAI evaluation.",

keywords = "Evaluation framework, Human-centred evaluation, XAI Evaluation",

author = "Ivania Donoso-Guzm{\'a}n and Jeroen Ooge and Denis Parra and Katrien Verbert",

note = "Publisher Copyright: {\textcopyright} 2023, The Author(s), under exclusive license to Springer Nature Switzerland AG.; 1st World Conference on eXplainable Artificial Intelligence, xAI 2023 ; Conference date: 26-07-2023 Through 28-07-2023",

year = "2023",

month = oct,

day = "21",

doi = "10.1007/978-3-031-44070-0_10",

language = "English",

isbn = "9783031440694",

series = "Communications in Computer and Information Science",

publisher = "Springer",

pages = "183--204",

editor = "Luca Longo",

booktitle = "Explainable Artificial Intelligence - 1st World Conference, xAI 2023, Proceedings",

}

Donoso-Guzmán, I, Ooge, J, Parra, D & Verbert, K 2023, Towards a Comprehensive Human-Centred Evaluation Framework for Explainable AI. in L Longo (ed.), Explainable Artificial Intelligence - 1st World Conference, xAI 2023, Proceedings. Communications in Computer and Information Science, vol. 1903 CCIS, Springer, pp. 183-204, 1st World Conference on eXplainable Artificial Intelligence, xAI 2023, Lisbon, Portugal, 26/07/23. https://doi.org/10.1007/978-3-031-44070-0_10

Towards a Comprehensive Human-Centred Evaluation Framework for Explainable AI. / Donoso-Guzmán, Ivania; Ooge, Jeroen; Parra, Denis et al.
Explainable Artificial Intelligence - 1st World Conference, xAI 2023, Proceedings. ed. / Luca Longo. Springer, 2023. p. 183-204 (Communications in Computer and Information Science; Vol. 1903 CCIS).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › Academic › peer-review

TY - GEN

T1 - Towards a Comprehensive Human-Centred Evaluation Framework for Explainable AI

AU - Donoso-Guzmán, Ivania

AU - Ooge, Jeroen

AU - Parra, Denis

AU - Verbert, Katrien

PY - 2023/10/21

Y1 - 2023/10/21

N2 - While research on explainable AI (XAI) is booming and explanation techniques have proven promising in many application domains, standardised human-centred evaluation procedures are still missing. In addition, current evaluation procedures do not assess XAI methods holistically in the sense that they do not treat explanations’ effects on humans as a complex user experience. To tackle this challenge, we propose to adapt the User-Centric Evaluation Framework used in recommender systems: we integrate explanation aspects, summarise explanation properties, indicate relations between them, and categorise metrics that measure these properties. With this comprehensive evaluation framework, we hope to contribute to the human-centred standardisation of XAI evaluation.

AB - While research on explainable AI (XAI) is booming and explanation techniques have proven promising in many application domains, standardised human-centred evaluation procedures are still missing. In addition, current evaluation procedures do not assess XAI methods holistically in the sense that they do not treat explanations’ effects on humans as a complex user experience. To tackle this challenge, we propose to adapt the User-Centric Evaluation Framework used in recommender systems: we integrate explanation aspects, summarise explanation properties, indicate relations between them, and categorise metrics that measure these properties. With this comprehensive evaluation framework, we hope to contribute to the human-centred standardisation of XAI evaluation.

KW - Evaluation framework

KW - Human-centred evaluation

KW - XAI Evaluation

UR - http://www.scopus.com/inward/record.url?scp=85175966305&partnerID=8YFLogxK

U2 - 10.1007/978-3-031-44070-0_10

DO - 10.1007/978-3-031-44070-0_10

M3 - Conference contribution

AN - SCOPUS:85175966305

SN - 9783031440694

T3 - Communications in Computer and Information Science

SP - 183

EP - 204

BT - Explainable Artificial Intelligence - 1st World Conference, xAI 2023, Proceedings

A2 - Longo, Luca

PB - Springer

T2 - 1st World Conference on eXplainable Artificial Intelligence, xAI 2023

Y2 - 26 July 2023 through 28 July 2023

ER -