Detecting Code Quality Issues in Pre-written Templates of Programming Tasks in Online Courses

Anastasiia Birillo; Elizaveta Artser; Yaroslav Golubev; Maria Tigina; Hieke Keuning; Nikolay Vyahhi; Timofey Bryksin

doi:10.1145/3587102.3588800

Detecting Code Quality Issues in Pre-written Templates of Programming Tasks in Online Courses

Anastasiia Birillo, Elizaveta Artser, Yaroslav Golubev, Maria Tigina, Hieke Keuning, Nikolay Vyahhi, Timofey Bryksin

Sub Software Technology for Learning and Teaching

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › Academic › peer-review

Abstract

In this work, we developed an algorithm for detecting code quality issues in the templates of online programming tasks, validated it, and conducted an empirical study on the dataset of student solutions. The algorithm consists of analyzing recurring unfixed issues in solutions of different students, matching them with the code of the template, and then filtering the results. Our manual validation on a subset of tasks demonstrated a precision of 80.8% and a recall of 73.3%. We used the algorithm on 415 Java tasks from the JetBrains Academy platform and discovered that as much as 14.7% of tasks have at least one issue in their template, thus making it harder for students to learn good code quality practices. We describe our results in detail, provide several motivating examples and specific cases, and share the feedback of the developers of the platform, who fixed 51 issues based on the output of our approach.

Original language	English
Title of host publication	ITiCSE 2023 - Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education
Publisher	Association for Computing Machinery
Pages	152-158
Number of pages	7
ISBN (Electronic)	9798400701382
DOIs	https://doi.org/10.1145/3587102.3588800
Publication status	Published - 29 Jun 2023
Event	28th Annual Conference on Innovation and Technology in Computer Science Education, ITiCSE 2023 - Turku, Finland Duration: 8 Jul 2023 → 12 Jul 2023

Publication series

Name	Annual Conference on Innovation and Technology in Computer Science Education, ITiCSE
Volume	1
ISSN (Print)	1942-647X

Conference

Conference	28th Annual Conference on Innovation and Technology in Computer Science Education, ITiCSE 2023
Country/Territory	Finland
City	Turku
Period	8/07/23 → 12/07/23

Bibliographical note

Publisher Copyright:
© 2023 ACM.

Keywords

code formatting
code quality
large-scale analysis
learning programming
MOOC
programming education
refactoring

Access to Document

10.1145/3587102.3588800

3587102.3588800Final published version, 1.54 MBLicence: Taverne

Cite this

Birillo, A., Artser, E., Golubev, Y., Tigina, M., Keuning, H., Vyahhi, N., & Bryksin, T. (2023). Detecting Code Quality Issues in Pre-written Templates of Programming Tasks in Online Courses. In ITiCSE 2023 - Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education (pp. 152-158). (Annual Conference on Innovation and Technology in Computer Science Education, ITiCSE; Vol. 1). Association for Computing Machinery. https://doi.org/10.1145/3587102.3588800

Birillo, Anastasiia ; Artser, Elizaveta ; Golubev, Yaroslav et al. / Detecting Code Quality Issues in Pre-written Templates of Programming Tasks in Online Courses. ITiCSE 2023 - Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education. Association for Computing Machinery, 2023. pp. 152-158 (Annual Conference on Innovation and Technology in Computer Science Education, ITiCSE).

@inproceedings{5dc24d5a61ac4b5fa1bbf2cd7afabdb5,

title = "Detecting Code Quality Issues in Pre-written Templates of Programming Tasks in Online Courses",

abstract = "In this work, we developed an algorithm for detecting code quality issues in the templates of online programming tasks, validated it, and conducted an empirical study on the dataset of student solutions. The algorithm consists of analyzing recurring unfixed issues in solutions of different students, matching them with the code of the template, and then filtering the results. Our manual validation on a subset of tasks demonstrated a precision of 80.8% and a recall of 73.3%. We used the algorithm on 415 Java tasks from the JetBrains Academy platform and discovered that as much as 14.7% of tasks have at least one issue in their template, thus making it harder for students to learn good code quality practices. We describe our results in detail, provide several motivating examples and specific cases, and share the feedback of the developers of the platform, who fixed 51 issues based on the output of our approach.",

keywords = "code formatting, code quality, large-scale analysis, learning programming, MOOC, programming education, refactoring",

author = "Anastasiia Birillo and Elizaveta Artser and Yaroslav Golubev and Maria Tigina and Hieke Keuning and Nikolay Vyahhi and Timofey Bryksin",

note = "Publisher Copyright: {\textcopyright} 2023 ACM.; 28th Annual Conference on Innovation and Technology in Computer Science Education, ITiCSE 2023 ; Conference date: 08-07-2023 Through 12-07-2023",

year = "2023",

month = jun,

day = "29",

doi = "10.1145/3587102.3588800",

language = "English",

series = "Annual Conference on Innovation and Technology in Computer Science Education, ITiCSE",

publisher = "Association for Computing Machinery",

pages = "152--158",

booktitle = "ITiCSE 2023 - Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education",

address = "United States",

}

Birillo, A, Artser, E, Golubev, Y, Tigina, M, Keuning, H, Vyahhi, N & Bryksin, T 2023, Detecting Code Quality Issues in Pre-written Templates of Programming Tasks in Online Courses. in ITiCSE 2023 - Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education. Annual Conference on Innovation and Technology in Computer Science Education, ITiCSE, vol. 1, Association for Computing Machinery, pp. 152-158, 28th Annual Conference on Innovation and Technology in Computer Science Education, ITiCSE 2023, Turku, Finland, 8/07/23. https://doi.org/10.1145/3587102.3588800

Detecting Code Quality Issues in Pre-written Templates of Programming Tasks in Online Courses. / Birillo, Anastasiia; Artser, Elizaveta; Golubev, Yaroslav et al.
ITiCSE 2023 - Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education. Association for Computing Machinery, 2023. p. 152-158 (Annual Conference on Innovation and Technology in Computer Science Education, ITiCSE; Vol. 1).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › Academic › peer-review

TY - GEN

T1 - Detecting Code Quality Issues in Pre-written Templates of Programming Tasks in Online Courses

AU - Birillo, Anastasiia

AU - Artser, Elizaveta

AU - Golubev, Yaroslav

AU - Tigina, Maria

AU - Keuning, Hieke

AU - Vyahhi, Nikolay

AU - Bryksin, Timofey

PY - 2023/6/29

Y1 - 2023/6/29

N2 - In this work, we developed an algorithm for detecting code quality issues in the templates of online programming tasks, validated it, and conducted an empirical study on the dataset of student solutions. The algorithm consists of analyzing recurring unfixed issues in solutions of different students, matching them with the code of the template, and then filtering the results. Our manual validation on a subset of tasks demonstrated a precision of 80.8% and a recall of 73.3%. We used the algorithm on 415 Java tasks from the JetBrains Academy platform and discovered that as much as 14.7% of tasks have at least one issue in their template, thus making it harder for students to learn good code quality practices. We describe our results in detail, provide several motivating examples and specific cases, and share the feedback of the developers of the platform, who fixed 51 issues based on the output of our approach.

AB - In this work, we developed an algorithm for detecting code quality issues in the templates of online programming tasks, validated it, and conducted an empirical study on the dataset of student solutions. The algorithm consists of analyzing recurring unfixed issues in solutions of different students, matching them with the code of the template, and then filtering the results. Our manual validation on a subset of tasks demonstrated a precision of 80.8% and a recall of 73.3%. We used the algorithm on 415 Java tasks from the JetBrains Academy platform and discovered that as much as 14.7% of tasks have at least one issue in their template, thus making it harder for students to learn good code quality practices. We describe our results in detail, provide several motivating examples and specific cases, and share the feedback of the developers of the platform, who fixed 51 issues based on the output of our approach.

KW - code formatting

KW - code quality

KW - large-scale analysis

KW - learning programming

KW - MOOC

KW - programming education

KW - refactoring

UR - http://www.scopus.com/inward/record.url?scp=85166186391&partnerID=8YFLogxK

U2 - 10.1145/3587102.3588800

DO - 10.1145/3587102.3588800

M3 - Conference contribution

AN - SCOPUS:85166186391

T3 - Annual Conference on Innovation and Technology in Computer Science Education, ITiCSE

SP - 152

EP - 158

BT - ITiCSE 2023 - Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education

PB - Association for Computing Machinery

T2 - 28th Annual Conference on Innovation and Technology in Computer Science Education, ITiCSE 2023

Y2 - 8 July 2023 through 12 July 2023

ER -

Birillo A, Artser E, Golubev Y, Tigina M, Keuning H, Vyahhi N et al. Detecting Code Quality Issues in Pre-written Templates of Programming Tasks in Online Courses. In ITiCSE 2023 - Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education. Association for Computing Machinery. 2023. p. 152-158. (Annual Conference on Innovation and Technology in Computer Science Education, ITiCSE). doi: 10.1145/3587102.3588800

Detecting Code Quality Issues in Pre-written Templates of Programming Tasks in Online Courses

Abstract

Publication series

Conference

Bibliographical note

Keywords

Access to Document

Other files and links

Fingerprint

Cite this