Analyzing Patterns of Conversational Breakdown in Human-Chatbot Customer Service Conversations

Anouck Braggaar; Florian Kunneman; Emiel van Miltenburg

doi:10.1007/978-3-031-88045-2_1

Analyzing Patterns of Conversational Breakdown in Human-Chatbot Customer Service Conversations

Anouck Braggaar^*, Florian Kunneman, Emiel van Miltenburg

^*Corresponding author for this work

Tilburg University

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › Academic › peer-review

Abstract

Many chatbots still struggle with correctly interpreting and responding to user enquiries. Therefore, it is important to figure out how and why chatbot-human conversations break down. In this study we analyzed features in user-utterances directly before a bot-initiated repair to determine their presence and prominence as possible predictors of conversational breakdowns. We used data from a real-life public transport customer service chatbot to demonstrate the errors that occur in actual deployed systems. The analysis shows that there are some features (such as commonness, outdated words, and unexpected words) that occur more often in utterances directly before a repair. Some features also correlate with each other and occur together, such as outdated words and subjectivity. By using feature analysis, many opportunities for improvement can be found either live (during the interaction) or afterwards.

Original language	English
Title of host publication	Chatbots and Human-Centered AI - 8th International Workshop, CONVERSATIONS 2024, Revised Selected Papers
Editors	Asbjørn Følstad, Symeon Papadopoulos, Theo Araujo, Effie L.-C. Law, Ewa Luger, Sebastian Hobert, Petter Bae Brandtzaeg
Publisher	Springer
Pages	3-22
Number of pages	20
ISBN (Electronic)	978-3-031-88045-2
ISBN (Print)	978-3-031-88044-5
DOIs	https://doi.org/10.1007/978-3-031-88045-2_1
Publication status	Published - 4 Apr 2025
Event	8th International Workshop on Chatbots and Human-Centered AI, CONVERSATIONS 2024 - Thessaloniki, Greece Duration: 4 Dec 2024 → 5 Dec 2024

Publication series

Name	Lecture Notes in Computer Science
Volume	15545 LNCS
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Conference

Conference	8th International Workshop on Chatbots and Human-Centered AI, CONVERSATIONS 2024
Country/Territory	Greece
City	Thessaloniki
Period	4/12/24 → 5/12/24

Bibliographical note

Publisher Copyright:
© The Author(s), under exclusive license to Springer Nature Switzerland AG 2025.

Keywords

Breakdowns
Chatbots
Customer service
Features analysis

Access to Document

10.1007/978-3-031-88045-2_1

Cite this

Braggaar, A., Kunneman, F., & van Miltenburg, E. (2025). Analyzing Patterns of Conversational Breakdown in Human-Chatbot Customer Service Conversations. In A. Følstad, S. Papadopoulos, T. Araujo, E. L.-C. Law, E. Luger, S. Hobert, & P. B. Brandtzaeg (Eds.), Chatbots and Human-Centered AI - 8th International Workshop, CONVERSATIONS 2024, Revised Selected Papers (pp. 3-22). (Lecture Notes in Computer Science; Vol. 15545 LNCS). Springer. https://doi.org/10.1007/978-3-031-88045-2_1

Braggaar, Anouck ; Kunneman, Florian ; van Miltenburg, Emiel. / Analyzing Patterns of Conversational Breakdown in Human-Chatbot Customer Service Conversations. Chatbots and Human-Centered AI - 8th International Workshop, CONVERSATIONS 2024, Revised Selected Papers. editor / Asbjørn Følstad ; Symeon Papadopoulos ; Theo Araujo ; Effie L.-C. Law ; Ewa Luger ; Sebastian Hobert ; Petter Bae Brandtzaeg. Springer, 2025. pp. 3-22 (Lecture Notes in Computer Science).

@inproceedings{31643b4ee39745109767f7dbb410ae09,

title = "Analyzing Patterns of Conversational Breakdown in Human-Chatbot Customer Service Conversations",

abstract = "Many chatbots still struggle with correctly interpreting and responding to user enquiries. Therefore, it is important to figure out how and why chatbot-human conversations break down. In this study we analyzed features in user-utterances directly before a bot-initiated repair to determine their presence and prominence as possible predictors of conversational breakdowns. We used data from a real-life public transport customer service chatbot to demonstrate the errors that occur in actual deployed systems. The analysis shows that there are some features (such as commonness, outdated words, and unexpected words) that occur more often in utterances directly before a repair. Some features also correlate with each other and occur together, such as outdated words and subjectivity. By using feature analysis, many opportunities for improvement can be found either live (during the interaction) or afterwards.",

keywords = "Breakdowns, Chatbots, Customer service, Features analysis",

author = "Anouck Braggaar and Florian Kunneman and {van Miltenburg}, Emiel",

note = "Publisher Copyright: {\textcopyright} The Author(s), under exclusive license to Springer Nature Switzerland AG 2025.; 8th International Workshop on Chatbots and Human-Centered AI, CONVERSATIONS 2024 ; Conference date: 04-12-2024 Through 05-12-2024",

year = "2025",

month = apr,

day = "4",

doi = "10.1007/978-3-031-88045-2_1",

language = "English",

isbn = "978-3-031-88044-5",

series = "Lecture Notes in Computer Science",

publisher = "Springer",

pages = "3--22",

editor = "Asbj{\o}rn F{\o}lstad and Symeon Papadopoulos and Theo Araujo and Law, {Effie L.-C.} and Ewa Luger and Sebastian Hobert and Brandtzaeg, {Petter Bae}",

booktitle = "Chatbots and Human-Centered AI - 8th International Workshop, CONVERSATIONS 2024, Revised Selected Papers",

}

Braggaar, A, Kunneman, F & van Miltenburg, E 2025, Analyzing Patterns of Conversational Breakdown in Human-Chatbot Customer Service Conversations. in A Følstad, S Papadopoulos, T Araujo, EL-C Law, E Luger, S Hobert & PB Brandtzaeg (eds), Chatbots and Human-Centered AI - 8th International Workshop, CONVERSATIONS 2024, Revised Selected Papers. Lecture Notes in Computer Science, vol. 15545 LNCS, Springer, pp. 3-22, 8th International Workshop on Chatbots and Human-Centered AI, CONVERSATIONS 2024, Thessaloniki, Greece, 4/12/24. https://doi.org/10.1007/978-3-031-88045-2_1

Analyzing Patterns of Conversational Breakdown in Human-Chatbot Customer Service Conversations. / Braggaar, Anouck; Kunneman, Florian; van Miltenburg, Emiel.
Chatbots and Human-Centered AI - 8th International Workshop, CONVERSATIONS 2024, Revised Selected Papers. ed. / Asbjørn Følstad; Symeon Papadopoulos; Theo Araujo; Effie L.-C. Law; Ewa Luger; Sebastian Hobert; Petter Bae Brandtzaeg. Springer, 2025. p. 3-22 (Lecture Notes in Computer Science; Vol. 15545 LNCS).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › Academic › peer-review

TY - GEN

T1 - Analyzing Patterns of Conversational Breakdown in Human-Chatbot Customer Service Conversations

AU - Braggaar, Anouck

AU - Kunneman, Florian

AU - van Miltenburg, Emiel

N1 - Publisher Copyright: © The Author(s), under exclusive license to Springer Nature Switzerland AG 2025.

PY - 2025/4/4

Y1 - 2025/4/4

N2 - Many chatbots still struggle with correctly interpreting and responding to user enquiries. Therefore, it is important to figure out how and why chatbot-human conversations break down. In this study we analyzed features in user-utterances directly before a bot-initiated repair to determine their presence and prominence as possible predictors of conversational breakdowns. We used data from a real-life public transport customer service chatbot to demonstrate the errors that occur in actual deployed systems. The analysis shows that there are some features (such as commonness, outdated words, and unexpected words) that occur more often in utterances directly before a repair. Some features also correlate with each other and occur together, such as outdated words and subjectivity. By using feature analysis, many opportunities for improvement can be found either live (during the interaction) or afterwards.

AB - Many chatbots still struggle with correctly interpreting and responding to user enquiries. Therefore, it is important to figure out how and why chatbot-human conversations break down. In this study we analyzed features in user-utterances directly before a bot-initiated repair to determine their presence and prominence as possible predictors of conversational breakdowns. We used data from a real-life public transport customer service chatbot to demonstrate the errors that occur in actual deployed systems. The analysis shows that there are some features (such as commonness, outdated words, and unexpected words) that occur more often in utterances directly before a repair. Some features also correlate with each other and occur together, such as outdated words and subjectivity. By using feature analysis, many opportunities for improvement can be found either live (during the interaction) or afterwards.

KW - Breakdowns

KW - Chatbots

KW - Customer service

KW - Features analysis

UR - http://www.scopus.com/inward/record.url?scp=105002906320&partnerID=8YFLogxK

U2 - 10.1007/978-3-031-88045-2_1

DO - 10.1007/978-3-031-88045-2_1

M3 - Conference contribution

AN - SCOPUS:105002906320

SN - 978-3-031-88044-5

T3 - Lecture Notes in Computer Science

SP - 3

EP - 22

BT - Chatbots and Human-Centered AI - 8th International Workshop, CONVERSATIONS 2024, Revised Selected Papers

A2 - Følstad, Asbjørn

A2 - Papadopoulos, Symeon

A2 - Araujo, Theo

A2 - Law, Effie L.-C.

A2 - Luger, Ewa

A2 - Hobert, Sebastian

A2 - Brandtzaeg, Petter Bae

PB - Springer

T2 - 8th International Workshop on Chatbots and Human-Centered AI, CONVERSATIONS 2024

Y2 - 4 December 2024 through 5 December 2024

ER -

Braggaar A, Kunneman F, van Miltenburg E. Analyzing Patterns of Conversational Breakdown in Human-Chatbot Customer Service Conversations. In Følstad A, Papadopoulos S, Araujo T, Law ELC, Luger E, Hobert S, Brandtzaeg PB, editors, Chatbots and Human-Centered AI - 8th International Workshop, CONVERSATIONS 2024, Revised Selected Papers. Springer. 2025. p. 3-22. (Lecture Notes in Computer Science). doi: 10.1007/978-3-031-88045-2_1