Exploring the Utility of Dutch Question Answering Datasets for Human Resource Contact Centres

Chaïm van Toledo; Marijn Schraagen; Friso Willem van Dijk; Matthieu Brinkhuis; Marco Spruit

doi:10.3390/info13110513

Exploring the Utility of Dutch Question Answering Datasets for Human Resource Contact Centres

Chaïm van Toledo, Marijn Schraagen, Friso Willem van Dijk, Matthieu Brinkhuis, Marco Spruit

Research output: Contribution to journal › Article › Academic › peer-review

Abstract

We explore the use case of question answering (QA) by a contact centre for 130,000 Dutch government employees in the domain of questions about human resources (HR). HR questions can be answered using personnel files or general documentation, with the latter being the focus of the current research. We created a Dutch HR QA dataset with over 300 questions in the format of the Squad 2.0 dataset, which distinguishes between answerable and unanswerable questions. We applied various BERT-based models, either directly or after finetuning on the new dataset. The F1-scores reached 0.47 for unanswerable questions and 1.0 for answerable questions depending on the topic; however, large variations in scores were observed. We conclude more data are needed to further improve the performance of this task.

Original language	English
Article number	513
Pages (from-to)	1-11
Journal	Information
Volume	13
Issue number	11
DOIs	https://doi.org/10.3390/info13110513
Publication status	Published - Nov 2022

Bibliographical note

Funding Information:
This research was funded by P-Direkt, Ministry of the Interior and Kingdom Relations, The Netherlands.

Publisher Copyright:
© 2022 by the authors.

Keywords

question and answering
dataset
Squad 2.0
question and answering dataset creation
question and answering Dutch
human resource dataset

Access to Document

10.3390/info13110513Licence: CC BY

information-13-00513Final published version, 725 KBLicence: CC BY

Cite this

@article{886c704286104138b9de958edbdc5dce,

title = "Exploring the Utility of Dutch Question Answering Datasets for Human Resource Contact Centres",

abstract = "We explore the use case of question answering (QA) by a contact centre for 130,000 Dutch government employees in the domain of questions about human resources (HR). HR questions can be answered using personnel files or general documentation, with the latter being the focus of the current research. We created a Dutch HR QA dataset with over 300 questions in the format of the Squad 2.0 dataset, which distinguishes between answerable and unanswerable questions. We applied various BERT-based models, either directly or after finetuning on the new dataset. The F1-scores reached 0.47 for unanswerable questions and 1.0 for answerable questions depending on the topic; however, large variations in scores were observed. We conclude more data are needed to further improve the performance of this task.",

keywords = "question and answering, dataset, Squad 2.0, question and answering dataset creation, question and answering Dutch, human resource dataset",

author = "{van Toledo}, Cha{\"i}m and Marijn Schraagen and {van Dijk}, {Friso Willem} and Matthieu Brinkhuis and Marco Spruit",

note = "Funding Information: This research was funded by P-Direkt, Ministry of the Interior and Kingdom Relations, The Netherlands. Publisher Copyright: {\textcopyright} 2022 by the authors.",

year = "2022",

month = nov,

doi = "10.3390/info13110513",

language = "English",

volume = "13",

pages = "1--11",

journal = "Information",

issn = "2078-2489",

publisher = "Multidisciplinary Digital Publishing Institute (MDPI)",

number = "11",

}

TY - JOUR

T1 - Exploring the Utility of Dutch Question Answering Datasets for Human Resource Contact Centres

AU - van Toledo, Chaïm

AU - Schraagen, Marijn

AU - van Dijk, Friso Willem

AU - Brinkhuis, Matthieu

AU - Spruit, Marco

PY - 2022/11

Y1 - 2022/11

N2 - We explore the use case of question answering (QA) by a contact centre for 130,000 Dutch government employees in the domain of questions about human resources (HR). HR questions can be answered using personnel files or general documentation, with the latter being the focus of the current research. We created a Dutch HR QA dataset with over 300 questions in the format of the Squad 2.0 dataset, which distinguishes between answerable and unanswerable questions. We applied various BERT-based models, either directly or after finetuning on the new dataset. The F1-scores reached 0.47 for unanswerable questions and 1.0 for answerable questions depending on the topic; however, large variations in scores were observed. We conclude more data are needed to further improve the performance of this task.

AB - We explore the use case of question answering (QA) by a contact centre for 130,000 Dutch government employees in the domain of questions about human resources (HR). HR questions can be answered using personnel files or general documentation, with the latter being the focus of the current research. We created a Dutch HR QA dataset with over 300 questions in the format of the Squad 2.0 dataset, which distinguishes between answerable and unanswerable questions. We applied various BERT-based models, either directly or after finetuning on the new dataset. The F1-scores reached 0.47 for unanswerable questions and 1.0 for answerable questions depending on the topic; however, large variations in scores were observed. We conclude more data are needed to further improve the performance of this task.

KW - question and answering

KW - dataset

KW - Squad 2.0

KW - question and answering dataset creation

KW - question and answering Dutch

KW - human resource dataset

UR - http://www.scopus.com/inward/record.url?scp=85141796951&partnerID=8YFLogxK

U2 - 10.3390/info13110513

DO - 10.3390/info13110513

M3 - Article

SN - 2078-2489

VL - 13

SP - 1

EP - 11

JO - Information

JF - Information

IS - 11

M1 - 513

ER -

Exploring the Utility of Dutch Question Answering Datasets for Human Resource Contact Centres

Abstract

Bibliographical note

Keywords

Access to Document

Other files and links

Fingerprint

Cite this