Blogs (4) >>

In the challenging field of introductory programming, high enrolments and failure rates drive us to explore tools and systems to enhance student outcomes, especially automated tools that scale to large cohorts. This paper presents and evaluates the dcc –help tool, an integration of a Large Language Model (LLM) into the Debugging C Compiler (DCC) to generate unique, novice-focused explanations tailored to each error. dcc –help prompts an LLM with contextual information of compile- and run-time error occurrences, including the source code, error location and standard compiler error message. The LLM is instructed to generate novice-focused, actionable error explanations and guidance, designed to help students understand and resolve problems without providing solutions. dcc –help was deployed to our CS1 and CS2 courses, with 2,565 students using the tool over 64,000 times in ten weeks. We analysed a subset of these error/explanation pairs to evaluate their properties, including conceptual correctness, relevancy, and overall quality. We found that the LLM-generated explanations were conceptually accurate in 90% of compile-time and 75% of run-time cases, but often disregarded the instruction not to provide solutions in code. Our findings, observations and reflections following deployment indicate that dcc –help provides novel opportunities for scaffolding students’ introduction to programming.

Fri 22 Mar

Displayed time zone: Pacific Time (US & Canada) change

15:45 - 17:00
LLMs - Error message and Coding strugglesPapers at Oregon Ballroom 204
Chair(s): Celine Latulipe University of Manitoba
15:45
25m
Talk
A Large Scale RCT on Effective Error Messages in CS1CER Best PaperGlobal
Papers
Sierra Wang Stanford University, John C. Mitchell Stanford University, Chris Piech Stanford University
DOI
16:10
25m
Talk
dcc --help: Transforming the Role of the Compiler by Generating Context-Aware Error Explanations with Large Language ModelsGlobal
Papers
Andrew Taylor University of New South Wales, Sydney, Alexandra Vassar University of New South Wales, Sydney, Jake Renzella University of New South Wales, Sydney, Hammond Pearce University of New South Wales, Sydney
DOI
16:35
25m
Talk
Exploring Novice Programmers' Testing Behavior: A first step to define coding struggle
Papers
Gabriel Silva de Oliveira North Carolina State University, Zhikai Gao North Carolina State University, Sarah Heckman North Carolina State University, Collin Lynch North Carolina State University
DOI