Design Choices in Crowdsourcing Discourse Relation Annotations: The Effect of Worker Selection and Training

Merel Scholman, Valentina Pyatkin, Frances Yung, Ido Dagan, Reut Tsarfaty, Vera Demberg

Research output: Chapter in Book/Report/Conference proceedingConference contributionAcademicpeer-review

Abstract

Obtaining linguistic annotation from novice crowdworkers is far from trivial. A case in point is the annotation of discourse relations, which is a complicated task. Recent methods have obtained promising results by extracting relation labels from either discourse connectives (DCs) or question-answer (QA) pairs that participants provide. The current contribution studies the effect of worker selection and training on the agreement on implicit relation labels between workers and gold labels, for both the DC and the QA method. In Study 1, workers were not specifically selected or trained, and the results show that there is much room for improvement. Study 2 shows that a combination of selection and training does lead to improved results, but the method is cost- and time-intensive. Study 3 shows that a selection-only approach is a viable alternative; it results in annotations of comparable quality compared to annotations from trained participants. The results generalized over both the DC and QA method and therefore indicate that a selection-only approach could also be effective for other crowdsourced discourse annotation tasks.
Original languageEnglish
Title of host publicationProceedings of the Thirteenth Language Resources and Evaluation Conference
PublisherEuropean Language Resources Association (ELRA)
Pages2148–2156
Publication statusPublished - 2022

Fingerprint

Dive into the research topics of 'Design Choices in Crowdsourcing Discourse Relation Annotations: The Effect of Worker Selection and Training'. Together they form a unique fingerprint.

Cite this