AMR Parsing is Far from Solved: GrAPES, the Granular AMR Parsing Evaluation Suite

Jonas Groschwitz; Shay Cohen; Lucia Donatelli; Meaghan Fowlie

doi:10.18653/v1/2023.emnlp-main.662

AMR Parsing is Far from Solved: GrAPES, the Granular AMR Parsing Evaluation Suite

Jonas Groschwitz, Shay Cohen, Lucia Donatelli, Meaghan Fowlie

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › Academic › peer-review

Abstract

We present the Granular AMR Parsing Evaluation Suite (GrAPES), a challenge set for Abstract Meaning Representation (AMR) parsing with accompanying evaluation metrics. AMR parsers now obtain high scores on the standard AMR evaluation metric Smatch, close to or even above reported inter-annotator agreement. But that does not mean that AMR parsing is solved; in fact, human evaluation in previous work indicates that current parsers still quite frequently make errors on node labels or graph structure that substantially distort sentence meaning. Here, we provide an evaluation suite that tests AMR parsers on a range of phenomena of practical, technical, and linguistic interest. Our 36 categories range from seen and unseen labels, to structural generalization, to coreference. GrAPES reveals in depth the abilities and shortcomings of current AMR parsers.

Original language	English
Title of host publication	Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing
Publisher	Association for Computational Linguistics
Pages	10728-10752
DOIs	https://doi.org/10.18653/v1/2023.emnlp-main.662
Publication status	Published - Dec 2023
Event	Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing - Singapore Duration: 1 Dec 2023 → 1 Dec 2023

Conference

Conference	Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing
Period	1/12/23 → 1/12/23

Access to Document

10.18653/v1/2023.emnlp-main.662Licence: CC BY

2023.emnlp-main.662Final published version, 294 KBLicence: CC BY

https://aclanthology.org/2023.emnlp-main.662

Cite this

@inproceedings{ed2f5d7435bf48baa47d781b99780f8f,

title = "AMR Parsing is Far from Solved: GrAPES, the Granular AMR Parsing Evaluation Suite",

abstract = "We present the Granular AMR Parsing Evaluation Suite (GrAPES), a challenge set for Abstract Meaning Representation (AMR) parsing with accompanying evaluation metrics. AMR parsers now obtain high scores on the standard AMR evaluation metric Smatch, close to or even above reported inter-annotator agreement. But that does not mean that AMR parsing is solved; in fact, human evaluation in previous work indicates that current parsers still quite frequently make errors on node labels or graph structure that substantially distort sentence meaning. Here, we provide an evaluation suite that tests AMR parsers on a range of phenomena of practical, technical, and linguistic interest. Our 36 categories range from seen and unseen labels, to structural generalization, to coreference. GrAPES reveals in depth the abilities and shortcomings of current AMR parsers.",

author = "Jonas Groschwitz and Shay Cohen and Lucia Donatelli and Meaghan Fowlie",

year = "2023",

month = dec,

doi = "10.18653/v1/2023.emnlp-main.662",

language = "English",

pages = "10728--10752",

booktitle = "Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing",

publisher = "Association for Computational Linguistics",

address = "United States",

note = "Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing ; Conference date: 01-12-2023 Through 01-12-2023",

}

Groschwitz, J, Cohen, S, Donatelli, L & Fowlie, M 2023, AMR Parsing is Far from Solved: GrAPES, the Granular AMR Parsing Evaluation Suite. in Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, pp. 10728-10752, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 1/12/23. https://doi.org/10.18653/v1/2023.emnlp-main.662

AMR Parsing is Far from Solved: GrAPES, the Granular AMR Parsing Evaluation Suite. / Groschwitz, Jonas; Cohen, Shay; Donatelli, Lucia et al.
Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 2023. p. 10728-10752.

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › Academic › peer-review

TY - GEN

T1 - AMR Parsing is Far from Solved: GrAPES, the Granular AMR Parsing Evaluation Suite

AU - Groschwitz, Jonas

AU - Cohen, Shay

AU - Donatelli, Lucia

AU - Fowlie, Meaghan

PY - 2023/12

Y1 - 2023/12

N2 - We present the Granular AMR Parsing Evaluation Suite (GrAPES), a challenge set for Abstract Meaning Representation (AMR) parsing with accompanying evaluation metrics. AMR parsers now obtain high scores on the standard AMR evaluation metric Smatch, close to or even above reported inter-annotator agreement. But that does not mean that AMR parsing is solved; in fact, human evaluation in previous work indicates that current parsers still quite frequently make errors on node labels or graph structure that substantially distort sentence meaning. Here, we provide an evaluation suite that tests AMR parsers on a range of phenomena of practical, technical, and linguistic interest. Our 36 categories range from seen and unseen labels, to structural generalization, to coreference. GrAPES reveals in depth the abilities and shortcomings of current AMR parsers.

AB - We present the Granular AMR Parsing Evaluation Suite (GrAPES), a challenge set for Abstract Meaning Representation (AMR) parsing with accompanying evaluation metrics. AMR parsers now obtain high scores on the standard AMR evaluation metric Smatch, close to or even above reported inter-annotator agreement. But that does not mean that AMR parsing is solved; in fact, human evaluation in previous work indicates that current parsers still quite frequently make errors on node labels or graph structure that substantially distort sentence meaning. Here, we provide an evaluation suite that tests AMR parsers on a range of phenomena of practical, technical, and linguistic interest. Our 36 categories range from seen and unseen labels, to structural generalization, to coreference. GrAPES reveals in depth the abilities and shortcomings of current AMR parsers.

U2 - 10.18653/v1/2023.emnlp-main.662

DO - 10.18653/v1/2023.emnlp-main.662

M3 - Conference contribution

SP - 10728

EP - 10752

BT - Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing

PB - Association for Computational Linguistics

T2 - Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing

Y2 - 1 December 2023 through 1 December 2023

ER -

AMR Parsing is Far from Solved: GrAPES, the Granular AMR Parsing Evaluation Suite

Abstract

Conference

Access to Document

Fingerprint

Cite this