Comparing the Accuracy of Automatic Scoring Solutions for a Text Comprehension Diagramming Intervention

Héctor Pijeira-Díaz, Jasper Snel, Sophia Braumann, Janneke van de Pol, Tamara van Gog, Anique B H de Bruin

Research output: Chapter in Book/Report/Conference proceedingConference contributionAcademicpeer-review


Students typically have great difficulty monitoring their comprehension of textual materials. Completing diagrams about causal relations in expository texts has been a successful intervention to enhance the accuracy of students’ reading comprehension judgments (ie, monitoring accuracy), although there is still room for improvement. Such judgments play a role in crucial self-regulated learning decisions that students make such as allocating time and effort, selecting content for restudy, and/or consulting additional sources. The automated scoring of students’ diagram content can provide a basis for strengthening the diagramming intervention with individual and simultaneous feedback to a high number of students. Leveraging an existing human-coded (correct and incorrect) dataset of 6000+ diagram answers (completed in Dutch by 700+ secondary students), we compared different automatic scoring solutions in terms of classification accuracy. Four computational linguistic models for Dutch were identified and tested in combination with four popular machine learning classification algorithms. The best solution reached 81% accuracy (ie, four out of five answers matched the human coding). Depending on the accuracy required for different applications, these results could be used for fully-or semiautomated scorings of students’ answers to generative activities used in reading comprehension interventions.
Original languageEnglish
Title of host publicationCompanion Proceedings of the 12th International Conference on Learning Analytics & Knowledge LAK22
Subtitle of host publicationLearning Analytics for Transition, Disruption and Social Chang
PublisherSociety for Learning Analytics Research
Publication statusPublished - Mar 2022
Event12th International Conference on Learning Analytics & Knowledge -
Duration: 21 Mar 202225 Mar 2022


Conference12th International Conference on Learning Analytics & Knowledge
Abbreviated titleLAK22
Internet address


Dive into the research topics of 'Comparing the Accuracy of Automatic Scoring Solutions for a Text Comprehension Diagramming Intervention'. Together they form a unique fingerprint.

Cite this