dcc --help: Transforming the Role of the Compiler by Generating Context-Aware Error Explanations with Large Language ModelsGlobal
In the challenging field of introductory programming, high enrolments and failure rates drive us to explore tools and systems to enhance student outcomes, especially automated tools that scale to large cohorts. This paper presents and evaluates the dcc –help tool, an integration of a Large Language Model (LLM) into the Debugging C Compiler (DCC) to generate unique, novice-focused explanations tailored to each error. dcc –help prompts an LLM with contextual information of compile- and run-time error occurrences, including the source code, error location and standard compiler error message. The LLM is instructed to generate novice-focused, actionable error explanations and guidance, designed to help students understand and resolve problems without providing solutions. dcc –help was deployed to our CS1 and CS2 courses, with 2,565 students using the tool over 64,000 times in ten weeks. We analysed a subset of these error/explanation pairs to evaluate their properties, including conceptual correctness, relevancy, and overall quality. We found that the LLM-generated explanations were conceptually accurate in 90% of compile-time and 75% of run-time cases, but often disregarded the instruction not to provide solutions in code. Our findings, observations and reflections following deployment indicate that dcc –help provides novel opportunities for scaffolding students’ introduction to programming.
Fri 22 MarDisplayed time zone: Pacific Time (US & Canada) change
15:45 - 17:00 | LLMs - Error message and Coding strugglesPapers at Oregon Ballroom 204 Chair(s): Celine Latulipe University of Manitoba | ||
15:45 25mTalk | A Large Scale RCT on Effective Error Messages in CS1Global Papers Sierra Wang Stanford University, John C. Mitchell Stanford University, Chris Piech Stanford University DOI | ||
16:10 25mTalk | dcc --help: Transforming the Role of the Compiler by Generating Context-Aware Error Explanations with Large Language ModelsGlobal Papers Andrew Taylor University of New South Wales, Sydney, Alexandra Vassar University of New South Wales, Sydney, Jake Renzella University of New South Wales, Sydney, Hammond Pearce University of New South Wales, Sydney DOI | ||
16:35 25mTalk | Exploring Novice Programmers' Testing Behavior: A first step to define coding struggle Papers Gabriel Silva de Oliveira North Carolina State University, Zhikai Gao North Carolina State University, Sarah Heckman North Carolina State University, Collin Lynch North Carolina State University DOI |