SemEval-2021 Task 12: Learning with Disagreements

Alexandra Uma; Tommaso Fornaciari; Anca Dumitrache; Tristan Miller; Jon Chamberlain; Barbara Plank; Edwin Simpson; Massimo Poesio

SemEval-2021 Task 12: Learning with Disagreements

Alexandra Uma, Tommaso Fornaciari, Anca Dumitrache, Tristan Miller, Jon Chamberlain, Barbara Plank, Edwin Simpson, Massimo Poesio

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › Academic › peer-review

Abstract

Disagreement between coders is ubiquitous in virtually all datasets annotated with human judgements in both natural language processing and computer vision. However, most supervised machine learning methods assume that a single preferred interpretation exists for each item, which is at best an idealization. The aim of the SemEval-2021 shared task on Learning with Disagreements (Le-wi-Di) was to provide a unified testing framework for methods for learning from data containing multiple and possibly contradictory annotations covering the best-known datasets containing information about disagreements for interpreting language and classifying images. In this paper we describe the shared task and its results.

Original language	English
Title of host publication	SemEval 2021 - 15th International Workshop on Semantic Evaluation, Proceedings of the Workshop
Editors	Alexis Palmer, Nathan Schneider, Natalie Schluter, Guy Emerson, Aurelie Herbelot, Xiaodan Zhu
Publisher	Association for Computational Linguistics
Pages	338-347
Number of pages	10
ISBN (Electronic)	9781954085701
Publication status	Published - 2021
Externally published	Yes
Event	15th International Workshop on Semantic Evaluation, SemEval 2021 - Virtual, Bangkok, Thailand Duration: 5 Aug 2021 → 6 Aug 2021

Publication series

Name	SemEval 2021 - 15th International Workshop on Semantic Evaluation, Proceedings of the Workshop

Conference

Conference	15th International Workshop on Semantic Evaluation, SemEval 2021
Country/Territory	Thailand
City	Virtual, Bangkok
Period	5/08/21 → 6/08/21

Bibliographical note

Publisher Copyright:
© 2021 Association for Computational Linguistics.

Funding

Alexandra Uma, Jon Chamberlain, and Massimo Poesio were partially supported by the DALI project, ERC Advanced Grant 695662. Tristan Miller was supported by the Austrian Science Fund (FWF) under project M 2625-N31. Barbara Plank is supported in part by the Independent Research Fund Denmark (DFF) grant 9131-00019B and 9063-00077B.

Funders	Funder number
Not added	695662
European Research Council
FWF Austrian Science Fund	M 2625-N31
Danmarks Frie Forskningsfond	9063-00077B, 9131-00019B

Cite this

Uma, A., Fornaciari, T., Dumitrache, A., Miller, T., Chamberlain, J., Plank, B., Simpson, E., & Poesio, M. (2021). SemEval-2021 Task 12: Learning with Disagreements. In A. Palmer, N. Schneider, N. Schluter, G. Emerson, A. Herbelot, & X. Zhu (Eds.), SemEval 2021 - 15th International Workshop on Semantic Evaluation, Proceedings of the Workshop (pp. 338-347). (SemEval 2021 - 15th International Workshop on Semantic Evaluation, Proceedings of the Workshop). Association for Computational Linguistics.

Uma, Alexandra ; Fornaciari, Tommaso ; Dumitrache, Anca et al. / SemEval-2021 Task 12 : Learning with Disagreements. SemEval 2021 - 15th International Workshop on Semantic Evaluation, Proceedings of the Workshop. editor / Alexis Palmer ; Nathan Schneider ; Natalie Schluter ; Guy Emerson ; Aurelie Herbelot ; Xiaodan Zhu. Association for Computational Linguistics, 2021. pp. 338-347 (SemEval 2021 - 15th International Workshop on Semantic Evaluation, Proceedings of the Workshop).

@inproceedings{3e6d30a141e5424fafd55e32370d86c5,

title = "SemEval-2021 Task 12: Learning with Disagreements",

abstract = "Disagreement between coders is ubiquitous in virtually all datasets annotated with human judgements in both natural language processing and computer vision. However, most supervised machine learning methods assume that a single preferred interpretation exists for each item, which is at best an idealization. The aim of the SemEval-2021 shared task on Learning with Disagreements (Le-wi-Di) was to provide a unified testing framework for methods for learning from data containing multiple and possibly contradictory annotations covering the best-known datasets containing information about disagreements for interpreting language and classifying images. In this paper we describe the shared task and its results.",

author = "Alexandra Uma and Tommaso Fornaciari and Anca Dumitrache and Tristan Miller and Jon Chamberlain and Barbara Plank and Edwin Simpson and Massimo Poesio",

note = "Publisher Copyright: {\textcopyright} 2021 Association for Computational Linguistics.; 15th International Workshop on Semantic Evaluation, SemEval 2021 ; Conference date: 05-08-2021 Through 06-08-2021",

year = "2021",

language = "English",

series = "SemEval 2021 - 15th International Workshop on Semantic Evaluation, Proceedings of the Workshop",

publisher = "Association for Computational Linguistics",

pages = "338--347",

editor = "Alexis Palmer and Nathan Schneider and Natalie Schluter and Guy Emerson and Aurelie Herbelot and Xiaodan Zhu",

booktitle = "SemEval 2021 - 15th International Workshop on Semantic Evaluation, Proceedings of the Workshop",

address = "United States",

}

Uma, A, Fornaciari, T, Dumitrache, A, Miller, T, Chamberlain, J, Plank, B, Simpson, E & Poesio, M 2021, SemEval-2021 Task 12: Learning with Disagreements. in A Palmer, N Schneider, N Schluter, G Emerson, A Herbelot & X Zhu (eds), SemEval 2021 - 15th International Workshop on Semantic Evaluation, Proceedings of the Workshop. SemEval 2021 - 15th International Workshop on Semantic Evaluation, Proceedings of the Workshop, Association for Computational Linguistics, pp. 338-347, 15th International Workshop on Semantic Evaluation, SemEval 2021, Virtual, Bangkok, Thailand, 5/08/21.

SemEval-2021 Task 12: Learning with Disagreements. / Uma, Alexandra; Fornaciari, Tommaso; Dumitrache, Anca et al.
SemEval 2021 - 15th International Workshop on Semantic Evaluation, Proceedings of the Workshop. ed. / Alexis Palmer; Nathan Schneider; Natalie Schluter; Guy Emerson; Aurelie Herbelot; Xiaodan Zhu. Association for Computational Linguistics, 2021. p. 338-347 (SemEval 2021 - 15th International Workshop on Semantic Evaluation, Proceedings of the Workshop).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › Academic › peer-review

TY - GEN

T1 - SemEval-2021 Task 12

T2 - 15th International Workshop on Semantic Evaluation, SemEval 2021

AU - Uma, Alexandra

AU - Fornaciari, Tommaso

AU - Dumitrache, Anca

AU - Miller, Tristan

AU - Chamberlain, Jon

AU - Plank, Barbara

AU - Simpson, Edwin

AU - Poesio, Massimo

PY - 2021

Y1 - 2021

N2 - Disagreement between coders is ubiquitous in virtually all datasets annotated with human judgements in both natural language processing and computer vision. However, most supervised machine learning methods assume that a single preferred interpretation exists for each item, which is at best an idealization. The aim of the SemEval-2021 shared task on Learning with Disagreements (Le-wi-Di) was to provide a unified testing framework for methods for learning from data containing multiple and possibly contradictory annotations covering the best-known datasets containing information about disagreements for interpreting language and classifying images. In this paper we describe the shared task and its results.

AB - Disagreement between coders is ubiquitous in virtually all datasets annotated with human judgements in both natural language processing and computer vision. However, most supervised machine learning methods assume that a single preferred interpretation exists for each item, which is at best an idealization. The aim of the SemEval-2021 shared task on Learning with Disagreements (Le-wi-Di) was to provide a unified testing framework for methods for learning from data containing multiple and possibly contradictory annotations covering the best-known datasets containing information about disagreements for interpreting language and classifying images. In this paper we describe the shared task and its results.

UR - http://www.scopus.com/inward/record.url?scp=85128526559&partnerID=8YFLogxK

M3 - Conference contribution

AN - SCOPUS:85128526559

T3 - SemEval 2021 - 15th International Workshop on Semantic Evaluation, Proceedings of the Workshop

SP - 338

EP - 347

BT - SemEval 2021 - 15th International Workshop on Semantic Evaluation, Proceedings of the Workshop

A2 - Palmer, Alexis

A2 - Schneider, Nathan

A2 - Schluter, Natalie

A2 - Emerson, Guy

A2 - Herbelot, Aurelie

A2 - Zhu, Xiaodan

PB - Association for Computational Linguistics

Y2 - 5 August 2021 through 6 August 2021

ER -

Uma A, Fornaciari T, Dumitrache A, Miller T, Chamberlain J, Plank B et al. SemEval-2021 Task 12: Learning with Disagreements. In Palmer A, Schneider N, Schluter N, Emerson G, Herbelot A, Zhu X, editors, SemEval 2021 - 15th International Workshop on Semantic Evaluation, Proceedings of the Workshop. Association for Computational Linguistics. 2021. p. 338-347. (SemEval 2021 - 15th International Workshop on Semantic Evaluation, Proceedings of the Workshop).

SemEval-2021 Task 12: Learning with Disagreements

Abstract

Publication series

Conference

Bibliographical note

Funding

Other files and links

Fingerprint

Cite this