Model solutions and properties for diagnosing student programs in Ask-Elle

Ask-Elle is an interactive tutor that supports the stepwise development
of simple functional programs. Using Ask-Elle students receive feedback
about whether or not they are on the right track, they can ask for a hint
when they are stuck, and get suggestions about how to refactor their program.
Our tutor generates this feedback from model solutions and properties
that a solution should satisfy. This paper studies the feasibility of using
model solutions together with the desired properties of solutions to analyse
the work of a student. It describes an experiment in which we analyse almost
3500 log entries from students using Ask-Elle to solve functional programming
exercises, to determine how many of these programs are diagnosed
correctly based on model solutions and the desired properties of solutions.
Ask-Elle manages to correctly diagnose 82.9% of the student programs. A
further analysis of the student programs and the diagnoses shows that adding
some reasonable model solutions, properties of model solutions, and
general program transformations would increase this percentage to 92.9%.