Semi-automated assessment of handwritten mathematics tasks: Atomic, reusable feedback for assessing student tests by teachers and exams by a group of assessors

Feedback is the most powerful engine of any learning process. In mathematics education, the possibilities to assess automatedly are thoroughly explored. However, students face difficulties expressing themselves mathematically on a computer and learning systems can often only assess the outcome, not the solving method. Research indicates that automated tests focus too much on procedural fluency at the expense of higher-order thinking questions. It takes much effort to develop digital tests, and teachers are sceptical of using automated assessments, meaning that paper-and-pencil tests still dominate mathematics classrooms. One of the characteristics of mathematical assessment is that wrong answers tend to exhibit patterns among the student population. Consequently, teachers often repeat their feedback and grades, bringing us to the idea of semi-automated feedback and assessment: by correcting handwritten tasks digitally, feedback can be saved and reused. This could lead to more elaborate feedback, time savings, and enhanced inter-rater reliability. Specifically, two semi-automatic assessment approaches were developed and studied.

In the first study, teachers write feedback for a student, and the computer saves it so that it can be reused when subsequent students make the same or similar mistakes. The concept of atomic feedback has been introduced to train teachers on how to write reusable feedback. Atomic feedback consists of a set of format requirements for mathematical feedback items, which has been shown to increase the reusability of feedback. A remarkable result was discovered during a crossover experiment with 45 mathematics teachers: the semi-automated approach led teachers to give significantly more feedback instead of saving time. Moreover, the teachers’ feedback with the semi-automatic tool did not always have better properties than classic pen-and-paper feedback.

The second study was conducted in collaboration with the Flemish Exam Commission. Their traditional grading method of handwritten mathematics exams was transformed into a semi-automated one called ‘checkbox grading.’ Every assessor receives a list of checkboxes, and they must tick those that apply to the student’s solution. Dependencies between these checkboxes can be set to ensure all assessors take the same path down the grading scheme. The system automatically calculates the grade and results in atomic feedback giving a detailed insight into what went wrong and how the grade was obtained. The approach requires more time for assessors and did not enhance inter-rater reliability compared to the traditional method (did not make it worse either). However, the resulting transparency and students’ feedback were highly valued. Moreover, students could easily understand the resulting feedback, even the lower-performing ones.