Next-Step Hint Generation for Introductory Programming Using Large Language Models

Lianne Roest; Hieke Keuning; Johan Jeuring

doi:10.1145/3636243.3636259

Next-Step Hint Generation for Introductory Programming Using Large Language Models

Lianne Roest, Hieke Keuning, Johan Jeuring

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › Academic › peer-review

Abstract

Large Language Models possess skills such as answering questions, writing essays or solving programming exercises. Since these models are easily accessible, researchers have investigated their capabilities and risks for programming education. This work explores how LLMs can contribute to programming education by supporting students with automated next-step hints. We investigate prompt practices that lead to effective next-step hints and use these insights to build our StAP-tutor. We evaluate this tutor by conducting an experiment with students, and performing expert assessments. Our findings show that most LLM-generated feedback messages describe one specific next step and are personalised to the student’s code and approach. However, the hints may contain misleading information and lack sufficient detail when students approach the end of the assignment. This work demonstrates the potential for LLM-generated feedback, but further research is required to explore its practical implementation.

Original language	English
Title of host publication	ACE 2024 - Proceedings of the 26th Australasian Computing Education Conference, Held in conjunction with
Subtitle of host publication	Australasian Computer Science Week
Editors	Nicole Herbert, Carolyn Seton
Publisher	Association for Computing Machinery
Pages	144-153
Number of pages	10
ISBN (Electronic)	9798400716195
ISBN (Print)	979-8-4007-1619-5
DOIs	https://doi.org/10.1145/3636243.3636259
Publication status	Published - Jan 2024

Publication series

Name	ACM International Conference Proceeding Series

Bibliographical note

Publisher Copyright:
© 2024 Copyright held by the owner/author(s). Publication rights licensed to ACM.

Keywords

Generative AI
Large Language Models
Next-step hints
automated feedback
learning programming

Access to Document

10.1145/3636243.3636259

Cite this

Roest, L., Keuning, H., & Jeuring, J. (2024). Next-Step Hint Generation for Introductory Programming Using Large Language Models. In N. Herbert, & C. Seton (Eds.), ACE 2024 - Proceedings of the 26th Australasian Computing Education Conference, Held in conjunction with: Australasian Computer Science Week (pp. 144-153). (ACM International Conference Proceeding Series). Association for Computing Machinery. https://doi.org/10.1145/3636243.3636259

Roest, Lianne ; Keuning, Hieke ; Jeuring, Johan. / Next-Step Hint Generation for Introductory Programming Using Large Language Models. ACE 2024 - Proceedings of the 26th Australasian Computing Education Conference, Held in conjunction with: Australasian Computer Science Week. editor / Nicole Herbert ; Carolyn Seton. Association for Computing Machinery, 2024. pp. 144-153 (ACM International Conference Proceeding Series).

@inproceedings{2be82794f6f04051acc253cdfe073626,

title = "Next-Step Hint Generation for Introductory Programming Using Large Language Models",

abstract = "Large Language Models possess skills such as answering questions, writing essays or solving programming exercises. Since these models are easily accessible, researchers have investigated their capabilities and risks for programming education. This work explores how LLMs can contribute to programming education by supporting students with automated next-step hints. We investigate prompt practices that lead to effective next-step hints and use these insights to build our StAP-tutor. We evaluate this tutor by conducting an experiment with students, and performing expert assessments. Our findings show that most LLM-generated feedback messages describe one specific next step and are personalised to the student{\textquoteright}s code and approach. However, the hints may contain misleading information and lack sufficient detail when students approach the end of the assignment. This work demonstrates the potential for LLM-generated feedback, but further research is required to explore its practical implementation.",

keywords = "Generative AI, Large Language Models, Next-step hints, automated feedback, learning programming",

author = "Lianne Roest and Hieke Keuning and Johan Jeuring",

note = "Publisher Copyright: {\textcopyright} 2024 Copyright held by the owner/author(s). Publication rights licensed to ACM.",

year = "2024",

month = jan,

doi = "10.1145/3636243.3636259",

language = "English",

isbn = "979-8-4007-1619-5",

series = "ACM International Conference Proceeding Series",

publisher = "Association for Computing Machinery",

pages = "144--153",

editor = "Herbert, {Nicole } and Seton, {Carolyn }",

booktitle = "ACE 2024 - Proceedings of the 26th Australasian Computing Education Conference, Held in conjunction with",

address = "United States",

}

Roest, L, Keuning, H & Jeuring, J 2024, Next-Step Hint Generation for Introductory Programming Using Large Language Models. in N Herbert & C Seton (eds), ACE 2024 - Proceedings of the 26th Australasian Computing Education Conference, Held in conjunction with: Australasian Computer Science Week. ACM International Conference Proceeding Series, Association for Computing Machinery, pp. 144-153. https://doi.org/10.1145/3636243.3636259

Next-Step Hint Generation for Introductory Programming Using Large Language Models. / Roest, Lianne; Keuning, Hieke ; Jeuring, Johan.
ACE 2024 - Proceedings of the 26th Australasian Computing Education Conference, Held in conjunction with: Australasian Computer Science Week. ed. / Nicole Herbert; Carolyn Seton. Association for Computing Machinery, 2024. p. 144-153 (ACM International Conference Proceeding Series).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › Academic › peer-review

TY - GEN

T1 - Next-Step Hint Generation for Introductory Programming Using Large Language Models

AU - Roest, Lianne

AU - Keuning, Hieke

AU - Jeuring, Johan

PY - 2024/1

Y1 - 2024/1

N2 - Large Language Models possess skills such as answering questions, writing essays or solving programming exercises. Since these models are easily accessible, researchers have investigated their capabilities and risks for programming education. This work explores how LLMs can contribute to programming education by supporting students with automated next-step hints. We investigate prompt practices that lead to effective next-step hints and use these insights to build our StAP-tutor. We evaluate this tutor by conducting an experiment with students, and performing expert assessments. Our findings show that most LLM-generated feedback messages describe one specific next step and are personalised to the student’s code and approach. However, the hints may contain misleading information and lack sufficient detail when students approach the end of the assignment. This work demonstrates the potential for LLM-generated feedback, but further research is required to explore its practical implementation.

AB - Large Language Models possess skills such as answering questions, writing essays or solving programming exercises. Since these models are easily accessible, researchers have investigated their capabilities and risks for programming education. This work explores how LLMs can contribute to programming education by supporting students with automated next-step hints. We investigate prompt practices that lead to effective next-step hints and use these insights to build our StAP-tutor. We evaluate this tutor by conducting an experiment with students, and performing expert assessments. Our findings show that most LLM-generated feedback messages describe one specific next step and are personalised to the student’s code and approach. However, the hints may contain misleading information and lack sufficient detail when students approach the end of the assignment. This work demonstrates the potential for LLM-generated feedback, but further research is required to explore its practical implementation.

KW - Generative AI

KW - Large Language Models

KW - Next-step hints

KW - automated feedback

KW - learning programming

UR - http://www.scopus.com/inward/record.url?scp=85182947216&partnerID=8YFLogxK

U2 - 10.1145/3636243.3636259

DO - 10.1145/3636243.3636259

M3 - Conference contribution

SN - 979-8-4007-1619-5

T3 - ACM International Conference Proceeding Series

SP - 144

EP - 153

BT - ACE 2024 - Proceedings of the 26th Australasian Computing Education Conference, Held in conjunction with

A2 - Herbert, Nicole

A2 - Seton, Carolyn

PB - Association for Computing Machinery

ER -

Roest L, Keuning H , Jeuring J. Next-Step Hint Generation for Introductory Programming Using Large Language Models. In Herbert N, Seton C, editors, ACE 2024 - Proceedings of the 26th Australasian Computing Education Conference, Held in conjunction with: Australasian Computer Science Week. Association for Computing Machinery. 2024. p. 144-153. (ACM International Conference Proceeding Series). doi: 10.1145/3636243.3636259

Next-Step Hint Generation for Introductory Programming Using Large Language Models

Abstract

Publication series

Bibliographical note

Keywords

Access to Document

Other files and links

Fingerprint

Cite this