Is there less agreement when the discourse is underspecified? Annotation of coherence relations in TED talks

Jet Hoek; Merel Scholman; Ted Sanders

Is there less agreement when the discourse is underspecified? Annotation of coherence relations in TED talks

ILS - Language and communication

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › Academic › peer-review

Abstract

When annotating coherence relations, interannotator agreement tends to be lower on implicit relations than on relations that are explicitly marked by means of a connective or
a cue phrase. This paper explores one possible explanation for this: the additional inferencing involved in interpreting implicit relations compared to explicit relations. If this is
the main source of disagreements, agreement
should be highly related to the specificity of
the connective. Using the CCR framework, we
annotated relations from TED talks that were
marked by a very specific marker, marked by
a highly ambiguous connective, or not marked
by means of a connective at all. We indeed
reached higher inter-annotator agreement on
explicit than on implicit relations. However,
agreement on underspecified relations was not
necessarily in between, which is what would
be expected if agreement on implicit relations
mainly suffers because annotators have less
specific instructions for inferring the relation.

Original language	English
Title of host publication	Proceedings of the First Workshop on Integrating Perspectives on Discourse Annotation
Editors	Kordula De Kuthy, Detmar Meurers
Publisher	Association for Computational Linguistics
Pages	1–6
Number of pages	6
Publication status	Published - 2021

Access to Document

2021.discann-1.1Final published version, 151 KBLicence: CC BY

https://aclanthology.org/2021.discann-1.1Licence: CC BY

Cite this

Hoek, J., Scholman, M., & Sanders, T. (2021). Is there less agreement when the discourse is underspecified? Annotation of coherence relations in TED talks. In K. De Kuthy, & D. Meurers (Eds.), Proceedings of the First Workshop on Integrating Perspectives on Discourse Annotation (pp. 1–6). Association for Computational Linguistics. https://aclanthology.org/2021.discann-1.1

@inproceedings{fc09a9156c124854a2ab064277521642,

title = "Is there less agreement when the discourse is underspecified? Annotation of coherence relations in TED talks",

abstract = "When annotating coherence relations, interannotator agreement tends to be lower on implicit relations than on relations that are explicitly marked by means of a connective ora cue phrase. This paper explores one possible explanation for this: the additional inferencing involved in interpreting implicit relations compared to explicit relations. If this isthe main source of disagreements, agreementshould be highly related to the specificity ofthe connective. Using the CCR framework, weannotated relations from TED talks that weremarked by a very specific marker, marked bya highly ambiguous connective, or not markedby means of a connective at all. We indeedreached higher inter-annotator agreement onexplicit than on implicit relations. However,agreement on underspecified relations was notnecessarily in between, which is what wouldbe expected if agreement on implicit relationsmainly suffers because annotators have lessspecific instructions for inferring the relation.",

author = "Jet Hoek and Merel Scholman and Ted Sanders",

year = "2021",

language = "English",

pages = "1–6",

editor = "{De Kuthy}, {Kordula } and Meurers, {Detmar }",

booktitle = "Proceedings of the First Workshop on Integrating Perspectives on Discourse Annotation",

publisher = "Association for Computational Linguistics",

address = "United States",

}

Is there less agreement when the discourse is underspecified? Annotation of coherence relations in TED talks. / Hoek, Jet; Scholman, Merel ; Sanders, Ted.
Proceedings of the First Workshop on Integrating Perspectives on Discourse Annotation. ed. / Kordula De Kuthy; Detmar Meurers. Association for Computational Linguistics, 2021. p. 1–6.

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › Academic › peer-review

TY - GEN

T1 - Is there less agreement when the discourse is underspecified? Annotation of coherence relations in TED talks

AU - Hoek, Jet

AU - Scholman, Merel

AU - Sanders, Ted

PY - 2021

Y1 - 2021

N2 - When annotating coherence relations, interannotator agreement tends to be lower on implicit relations than on relations that are explicitly marked by means of a connective ora cue phrase. This paper explores one possible explanation for this: the additional inferencing involved in interpreting implicit relations compared to explicit relations. If this isthe main source of disagreements, agreementshould be highly related to the specificity ofthe connective. Using the CCR framework, weannotated relations from TED talks that weremarked by a very specific marker, marked bya highly ambiguous connective, or not markedby means of a connective at all. We indeedreached higher inter-annotator agreement onexplicit than on implicit relations. However,agreement on underspecified relations was notnecessarily in between, which is what wouldbe expected if agreement on implicit relationsmainly suffers because annotators have lessspecific instructions for inferring the relation.

AB - When annotating coherence relations, interannotator agreement tends to be lower on implicit relations than on relations that are explicitly marked by means of a connective ora cue phrase. This paper explores one possible explanation for this: the additional inferencing involved in interpreting implicit relations compared to explicit relations. If this isthe main source of disagreements, agreementshould be highly related to the specificity ofthe connective. Using the CCR framework, weannotated relations from TED talks that weremarked by a very specific marker, marked bya highly ambiguous connective, or not markedby means of a connective at all. We indeedreached higher inter-annotator agreement onexplicit than on implicit relations. However,agreement on underspecified relations was notnecessarily in between, which is what wouldbe expected if agreement on implicit relationsmainly suffers because annotators have lessspecific instructions for inferring the relation.

M3 - Conference contribution

SP - 1

EP - 6

BT - Proceedings of the First Workshop on Integrating Perspectives on Discourse Annotation

A2 - De Kuthy, Kordula

A2 - Meurers, Detmar

PB - Association for Computational Linguistics

ER -

Is there less agreement when the discourse is underspecified? Annotation of coherence relations in TED talks

Abstract

Access to Document

Fingerprint

Cite this