The Universal Anaphora Scorer

Juntao Yu; Sopan Khosla; Nafise Sadat Moosavi; Silviu Paun; Sameer Pradhan; Massimo Poesio

The Universal Anaphora Scorer

Juntao Yu
, Sopan Khosla^*
, Nafise Sadat Moosavi
, Silviu Paun
, Sameer Pradhan
, Massimo Poesio

^*Corresponding author for this work

Sub Natural Language Processing

University of Essex
Carnegie Mellon University
Amazon.com, Inc.
University of Sheffield
Queen Mary University of London
University of Pennsylvania
cemantix.org

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › Academic › peer-review

Abstract

The aim of the Universal Anaphora initiative is to push forward the state of the art in anaphora and anaphora resolution by expanding the aspects of anaphoric interpretation which are or can be reliably annotated in anaphoric corpora, producing unified standards to annotate and encode these annotations, deliver datasets encoded according to these standards, and developing methods for evaluating models carrying out this type of interpretation. Such expansion of the scope of anaphora resolution requires a comparable expansion of the scope of the scorers used to evaluate this work. In this paper, we introduce an extended version of the Reference Coreference Scorer (Pradhan et al., 2014) that can be used to evaluate the extended range of anaphoric interpretation included in the current Universal Anaphora proposal. The UA scorer supports the evaluation of identity anaphora resolution and of bridging reference resolution, for which scorers already existed but not integrated in a single package. It also supports the evaluation of split antecedent anaphora and discourse deixis, for which no tools existed. The proposed approach to the evaluation of split antecedent anaphora is entirely novel; the proposed approach to the evaluation of discourse deixis leverages the encoding of discourse deixis proposed in Universal Anaphora to enable the use for discourse deixis of the same metrics already used for identity anaphora. The scorer was tested in the recent CODI-CRAC 2021 Shared Task on Anaphora Resolution in Dialogues.

Original language	English
Title of host publication	2022 Language Resources and Evaluation Conference, LREC 2022
Editors	Nicoletta Calzolari, Frederic Bechet, Philippe Blache, Khalid Choukri, Christopher Cieri, Thierry Declerck, Sara Goggi, Hitoshi Isahara, Bente Maegaard, Joseph Mariani, Helene Mazo, Jan Odijk, Stelios Piperidis
Publisher	European Language Resources Association (ELRA)
Pages	4873-4883
Number of pages	11
ISBN (Electronic)	9791095546726
Publication status	Published - Jun 2022
Event	13th International Conference on Language Resources and Evaluation Conference, LREC 2022 - Marseille, France Duration: 20 Jun 2022 → 25 Jun 2022

Publication series

Name	2022 Language Resources and Evaluation Conference, LREC 2022

Conference

Conference	13th International Conference on Language Resources and Evaluation Conference, LREC 2022
Country/Territory	France
City	Marseille
Period	20/06/22 → 25/06/22

Bibliographical note

Publisher Copyright:
© European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0.

Funding

The work of Juntao Yu, Silviu Paun and Massimo Poesio was funded by the DALI project, ERC Grant 695662.

Funders	Funder number
Dali University
Engineering Research Centers	695662

Keywords

Anaphora Resolution
Evaluation
Univeral Anaphora

Access to Document

2022.lrec-1.521Final published version, 278 KBLicence: CC BY-NC

https://aclanthology.org/2022.lrec-1.521Licence: CC BY-NC

Cite this

Yu, J., Khosla, S., Moosavi, N. S., Paun, S., Pradhan, S., & Poesio, M. (2022). The Universal Anaphora Scorer. In N. Calzolari, F. Bechet, P. Blache, K. Choukri, C. Cieri, T. Declerck, S. Goggi, H. Isahara, B. Maegaard, J. Mariani, H. Mazo, J. Odijk, & S. Piperidis (Eds.), 2022 Language Resources and Evaluation Conference, LREC 2022 (pp. 4873-4883). (2022 Language Resources and Evaluation Conference, LREC 2022). European Language Resources Association (ELRA). https://aclanthology.org/2022.lrec-1.521

Yu, Juntao ; Khosla, Sopan ; Moosavi, Nafise Sadat et al. / The Universal Anaphora Scorer. 2022 Language Resources and Evaluation Conference, LREC 2022. editor / Nicoletta Calzolari ; Frederic Bechet ; Philippe Blache ; Khalid Choukri ; Christopher Cieri ; Thierry Declerck ; Sara Goggi ; Hitoshi Isahara ; Bente Maegaard ; Joseph Mariani ; Helene Mazo ; Jan Odijk ; Stelios Piperidis. European Language Resources Association (ELRA), 2022. pp. 4873-4883 (2022 Language Resources and Evaluation Conference, LREC 2022).

@inproceedings{dc7434761ba1407399f771b3207e9525,

title = "The Universal Anaphora Scorer",

abstract = "The aim of the Universal Anaphora initiative is to push forward the state of the art in anaphora and anaphora resolution by expanding the aspects of anaphoric interpretation which are or can be reliably annotated in anaphoric corpora, producing unified standards to annotate and encode these annotations, deliver datasets encoded according to these standards, and developing methods for evaluating models carrying out this type of interpretation. Such expansion of the scope of anaphora resolution requires a comparable expansion of the scope of the scorers used to evaluate this work. In this paper, we introduce an extended version of the Reference Coreference Scorer (Pradhan et al., 2014) that can be used to evaluate the extended range of anaphoric interpretation included in the current Universal Anaphora proposal. The UA scorer supports the evaluation of identity anaphora resolution and of bridging reference resolution, for which scorers already existed but not integrated in a single package. It also supports the evaluation of split antecedent anaphora and discourse deixis, for which no tools existed. The proposed approach to the evaluation of split antecedent anaphora is entirely novel; the proposed approach to the evaluation of discourse deixis leverages the encoding of discourse deixis proposed in Universal Anaphora to enable the use for discourse deixis of the same metrics already used for identity anaphora. The scorer was tested in the recent CODI-CRAC 2021 Shared Task on Anaphora Resolution in Dialogues.",

keywords = "Anaphora Resolution, Evaluation, Univeral Anaphora",

author = "Juntao Yu and Sopan Khosla and Moosavi, \{Nafise Sadat\} and Silviu Paun and Sameer Pradhan and Massimo Poesio",

note = "Publisher Copyright: {\textcopyright} European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0.; 13th International Conference on Language Resources and Evaluation Conference, LREC 2022 ; Conference date: 20-06-2022 Through 25-06-2022",

year = "2022",

month = jun,

language = "English",

series = "2022 Language Resources and Evaluation Conference, LREC 2022",

publisher = "European Language Resources Association (ELRA)",

pages = "4873--4883",

editor = "Nicoletta Calzolari and Frederic Bechet and Philippe Blache and Khalid Choukri and Christopher Cieri and Thierry Declerck and Sara Goggi and Hitoshi Isahara and Bente Maegaard and Joseph Mariani and Helene Mazo and Jan Odijk and Stelios Piperidis",

booktitle = "2022 Language Resources and Evaluation Conference, LREC 2022",

}

Yu, J, Khosla, S, Moosavi, NS, Paun, S, Pradhan, S & Poesio, M 2022, The Universal Anaphora Scorer. in N Calzolari, F Bechet, P Blache, K Choukri, C Cieri, T Declerck, S Goggi, H Isahara, B Maegaard, J Mariani, H Mazo, J Odijk & S Piperidis (eds), 2022 Language Resources and Evaluation Conference, LREC 2022. 2022 Language Resources and Evaluation Conference, LREC 2022, European Language Resources Association (ELRA), pp. 4873-4883, 13th International Conference on Language Resources and Evaluation Conference, LREC 2022, Marseille, France, 20/06/22. <https://aclanthology.org/2022.lrec-1.521>

The Universal Anaphora Scorer. / Yu, Juntao; Khosla, Sopan; Moosavi, Nafise Sadat et al.
2022 Language Resources and Evaluation Conference, LREC 2022. ed. / Nicoletta Calzolari; Frederic Bechet; Philippe Blache; Khalid Choukri; Christopher Cieri; Thierry Declerck; Sara Goggi; Hitoshi Isahara; Bente Maegaard; Joseph Mariani; Helene Mazo; Jan Odijk; Stelios Piperidis. European Language Resources Association (ELRA), 2022. p. 4873-4883 (2022 Language Resources and Evaluation Conference, LREC 2022).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › Academic › peer-review

TY - GEN

T1 - The Universal Anaphora Scorer

AU - Yu, Juntao

AU - Khosla, Sopan

AU - Moosavi, Nafise Sadat

AU - Paun, Silviu

AU - Pradhan, Sameer

AU - Poesio, Massimo

N1 - Publisher Copyright: © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0.

PY - 2022/6

Y1 - 2022/6

N2 - The aim of the Universal Anaphora initiative is to push forward the state of the art in anaphora and anaphora resolution by expanding the aspects of anaphoric interpretation which are or can be reliably annotated in anaphoric corpora, producing unified standards to annotate and encode these annotations, deliver datasets encoded according to these standards, and developing methods for evaluating models carrying out this type of interpretation. Such expansion of the scope of anaphora resolution requires a comparable expansion of the scope of the scorers used to evaluate this work. In this paper, we introduce an extended version of the Reference Coreference Scorer (Pradhan et al., 2014) that can be used to evaluate the extended range of anaphoric interpretation included in the current Universal Anaphora proposal. The UA scorer supports the evaluation of identity anaphora resolution and of bridging reference resolution, for which scorers already existed but not integrated in a single package. It also supports the evaluation of split antecedent anaphora and discourse deixis, for which no tools existed. The proposed approach to the evaluation of split antecedent anaphora is entirely novel; the proposed approach to the evaluation of discourse deixis leverages the encoding of discourse deixis proposed in Universal Anaphora to enable the use for discourse deixis of the same metrics already used for identity anaphora. The scorer was tested in the recent CODI-CRAC 2021 Shared Task on Anaphora Resolution in Dialogues.

AB - The aim of the Universal Anaphora initiative is to push forward the state of the art in anaphora and anaphora resolution by expanding the aspects of anaphoric interpretation which are or can be reliably annotated in anaphoric corpora, producing unified standards to annotate and encode these annotations, deliver datasets encoded according to these standards, and developing methods for evaluating models carrying out this type of interpretation. Such expansion of the scope of anaphora resolution requires a comparable expansion of the scope of the scorers used to evaluate this work. In this paper, we introduce an extended version of the Reference Coreference Scorer (Pradhan et al., 2014) that can be used to evaluate the extended range of anaphoric interpretation included in the current Universal Anaphora proposal. The UA scorer supports the evaluation of identity anaphora resolution and of bridging reference resolution, for which scorers already existed but not integrated in a single package. It also supports the evaluation of split antecedent anaphora and discourse deixis, for which no tools existed. The proposed approach to the evaluation of split antecedent anaphora is entirely novel; the proposed approach to the evaluation of discourse deixis leverages the encoding of discourse deixis proposed in Universal Anaphora to enable the use for discourse deixis of the same metrics already used for identity anaphora. The scorer was tested in the recent CODI-CRAC 2021 Shared Task on Anaphora Resolution in Dialogues.

KW - Anaphora Resolution

KW - Evaluation

KW - Univeral Anaphora

UR - https://www.scopus.com/pages/publications/85144345380

M3 - Conference contribution

AN - SCOPUS:85144345380

T3 - 2022 Language Resources and Evaluation Conference, LREC 2022

SP - 4873

EP - 4883

BT - 2022 Language Resources and Evaluation Conference, LREC 2022

A2 - Calzolari, Nicoletta

A2 - Bechet, Frederic

A2 - Blache, Philippe

A2 - Choukri, Khalid

A2 - Cieri, Christopher

A2 - Declerck, Thierry

A2 - Goggi, Sara

A2 - Isahara, Hitoshi

A2 - Maegaard, Bente

A2 - Mariani, Joseph

A2 - Mazo, Helene

A2 - Odijk, Jan

A2 - Piperidis, Stelios

PB - European Language Resources Association (ELRA)

T2 - 13th International Conference on Language Resources and Evaluation Conference, LREC 2022

Y2 - 20 June 2022 through 25 June 2022

ER -

Yu J, Khosla S, Moosavi NS, Paun S, Pradhan S, Poesio M. The Universal Anaphora Scorer. In Calzolari N, Bechet F, Blache P, Choukri K, Cieri C, Declerck T, Goggi S, Isahara H, Maegaard B, Mariani J, Mazo H, Odijk J, Piperidis S, editors, 2022 Language Resources and Evaluation Conference, LREC 2022. European Language Resources Association (ELRA). 2022. p. 4873-4883. (2022 Language Resources and Evaluation Conference, LREC 2022).

The Universal Anaphora Scorer

Abstract

Publication series

Conference

Bibliographical note

Funding

Keywords

Access to Document

Other files and links

Fingerprint

Cite this