Evaluating GPT-4 Code Generation as a Grading Mechanism for “Explain-in-Plain-English” Questions (SIGCSE TS 2024 - Posters) - SIGCSE TS 2024

Wed 20 - Sat 23 March 2024 Portland, Oregon, United States

Who

David Smith, Craig Zilles

Track

SIGCSE TS 2024 Posters

Time Zone

The program is currently displayed in (GMT-07:00) Pacific Time (US & Canada).

Use conference time zone: (GMT-07:00) Pacific Time (US & Canada)Select other time zone

The GMT offsets shown reflect the offsets at the moment of the conference.

Time Band

By setting a time band, the program will dim events that are outside this time window. This is useful for (virtual) conferences with a continuous program (with repeated sessions).
The time band will also limit the events that are included in the personal iCalendar subscription service.

Display full programSpecify a time band

When

Sat 23 Mar 2024 10:00 - 12:00 at Exhibit Hall E - Posters - Posters 4 Chair(s): Meghan Allen, Ashish Aggarwal

Abstract

The ability to “Explain in plain English” (EiPE) is a critical skill for students in introductory programming skills to develop. However,evaluating this skill has been challenging as manual grading is time consuming and not easily automated. Constructing an effective prompt for a language model, such as OpenAI’s GPT-4, to generate code that achieves a specified goal bears a striking resemblance to the skill of EiPE. In this paper, we explore the potential of using test cases run on code generated by GPT-4 from students’ EiPE responses as a grading mechanism for EiPE questions. We applied this proposed grading method to a corpus of EiPE responses collected from past exams, then measured agreement between the results of this grading method and human graders. Overall, we find moderate agreement between the human raters and the results of the unittests run on the generated code. This appears to be attributable toGPT-4’s code generation being more lenient than human graders on low-level descriptions of code.

David Smith

University of Illinois at Urbana-Champaign

United States

Craig Zilles

University of Illinois at Urbana-Champaign

United States

Time Zone

The program is currently displayed in (GMT-07:00) Pacific Time (US & Canada).

Use conference time zone: (GMT-07:00) Pacific Time (US & Canada)Select other time zone

The GMT offsets shown reflect the offsets at the moment of the conference.

Time Band

By setting a time band, the program will dim events that are outside this time window. This is useful for (virtual) conferences with a continuous program (with repeated sessions).
The time band will also limit the events that are included in the personal iCalendar subscription service.

Display full programSpecify a time band

Session Program

Sat 23 Mar
Displayed time zone: Pacific Time (US & Canada) change

	10:00 - 12:00	Posters 4Posters at Exhibit Hall E - Posters Chair(s): Meghan Allen University of British Columbia, Ashish Aggarwal University of Florida

	10:00 2h Talk		Enhancing Code Tracing Question Generation with Refined Prompts in Large Language Models Posters Aysa Xuemo Fan University of Illinois Urbana-Champaign, Rully Agus Hendrawan University of Pittsburgh, Yang Shi North Carolina State University, Qianou Ma Carnegie Mellon University
	10:00 2h Talk		Leveraging Kotter's 8 Stage Model of Organizational Change to Understand Broadening Participation in Computing Posters Kari George University of Illinois at Urbana-Champaign, Max Fowler University of Illinois, Vidushi Ojha University of Illinois at Urbana-Champaign, Morgan Fong University of Illinois, Urbana-Champaign, Kathleen Isenegger University of Illinois at Urbana-Champaign, Christopher Perdriau University of Illinois at Urbana-Champaign, Mariam Saffar Perez University of Illinois at Urbana-Champaign, Yael Gertner University of Illinois Urbana-Champaign, Colleen M. Lewis University of Illinois at Urbana-Champaign
	10:00 2h Talk		Enhancing Formative Feedback at Scale with the Intelligent Feedback AssistantGlobal Posters Rifa Jamal University of New South Wales, Sydney, Jake Renzella University of New South Wales, Sydney
	10:00 2h Talk		Using Computational Thinking to Learn Creative Writing and Performance Arts: summer experiences of story creation projects Posters Ruth Torres Castillo New Mexico State University, Tomas Valdez New Mexico State University, Enrico Pontelli , Horacio Gonzalez New Mexico State University, Erick Nevarez New Mexico State University, Raena Cota New Mexico State University
	10:00 2h Talk		Restorying with AI Art among Latinx Elementary Students Posters Leiny Garcia University of California, Irvine, Santiago Ojeda-Ramirez University of California, Irvine, Mark Warschauer University of California, Irvine
	10:00 2h Talk		Exploring AIFORGOOD Summer Camp Curriculum to Foster Middle School Students' Understanding of Artificial IntelligenceK12 Posters Kwon Kyungbin Indiana University Bloomington, Keunjae Kim Indiana University Bloomington, Anne Leftwich Indiana University, Krista Glazewski North Carolina State University, Matthew Brown Indiana University Bloomington, Haesol Bae University at Albany, Tina Closser Indiana University Bloomington
	10:00 2h Talk		Immersing Undergraduates in Interdisciplinary Course Collaborations Posters S. Monisha Pulimood The College of New Jersey, Diane C. Bates The College of New Jersey, Kim Pearson The College of New Jersey
	10:00 2h Talk		Using Generative AI to Support PK-12 Teaching and Learning: Developing Sample Lessons and More Posters Pati Ruiz Digital Promise, Alessandra Rangel Digital Promise, Merijke Coenraad Digital Promise
	10:00 2h Talk		Integrating & Implementing K-12 Computing Pathways across Six School Districts—Challenges & Opportunities K12 Posters Quinn Burke Digital Promise Global, Merijke Coenraad Digital Promise, Alessandra Rangel Digital Promise, Kyle Dunbar Digital Promise Global, Kelly Mills Digital Promise Global
	10:00 2h Talk		Designing Create Tech: A Discussion of Previous Camp Iterations and Outcomes Posters Charlotte Avery University of Maryland, College Park
	10:00 2h Talk		Parallel Islands: A Parallel Computing Educational Video Game Posters Melissa Cameron Virginia Tech, Margaret Ellis Virginia Tech, Dimitrios Nikolopoulos Virginia Tech
	10:00 2h Talk		Perfect Snake: Exploring Algorithms Through the Snake Game Posters William Weinberger University of Southern California, Jingsai Liang Westminster University, Xi Chen Utah Valley University
	10:00 2h Talk		Effects of Prior Academic Experience in Introductory Level Data Science Course Posters Noshaba Bhalli University of Maryland, Baltimore County, Vandana P. Janeja University of Maryland, Baltimore County, David Harding UC Berkeley
	10:00 2h Talk		CT Readiness for All: Studying a Framework for Supporting Schoolwide Computational Thinking Integration Across Elementary Curricula K12 Posters Heather Sherwood Education Development Center, Babette Moeller Education Development Center, Cheri Fancsali Research Alliance for NYC Schools at NYU
	10:00 2h Talk		A Case Study of Taking AP Computer Science Principles (AP CSP): A Student’s Perspective Posters Sarah Cameron University of West Florida, Tony Pham University of West Florida, Sikha Bagui University of West Florida
	10:00 2h Talk		Advice for the First Time Hardware CTF Organizer Posters Chad Mourning Ohio University
	10:00 2h Talk		Integrating Personalized Parsons Problems with Multi-Level Textual Explanations to Scaffold Code Writing Posters Xinying Hou University of Michigan, Barbara Ericson University of Michigan, Xu Wang University of Michigan
	10:00 2h Talk		Reimagining CS Courses for High School StudentsK12 Posters Julie Smith Institute for Advancing Computing Education, Bryan Twarek Computer Science Teachers' Association, Monica McGill Institute for Advancing Computing Education
	10:00 2h Talk		Designing Theory of Computing Backwards Posters Ryan Dougherty United States Military Academy
	10:00 2h Talk		Reducing Barriers to Entry by Removing Prerequisites for a CS1 Introductory Programming Course Posters Udayan Das Saint Mary's College of California, Chris Fulton Loyola University Chicago
	10:00 2h Talk		Understanding the Leadership Structure and Mentoring Model of an Extracurricular Robotics Team: Key Findings from a Case Study Posters Hitender K Oswal University of Washington, Jonathan Wick Western Washington University, Seth Tandon University of Southern California, Ashley Brewster Science and Math Institute, Sushil K Oswal University of Washington
	10:00 2h Talk		Style Anomalies Can Suggest Cheating in CS1 Programs Posters Benjamin Denzler University of California, Riverside, Frank Vahid UC Riverside / zyBooks, Ashley Pang UC Riverside
	10:00 2h Talk		An Experience Report on Teaching a Large Introductory Programming Course in Hybrid ModeGlobal Posters Shyama Sastha Krishnamoorthy Srinivasan IIIT-Delhi, Avi Gupta Indraprastha Institute of Information Technology Delhi, Pankaj Jalote IIT Delhi
	10:00 2h Talk		Supporting Student Engagement in K-12 AI Education with a Card Game Construction Toolkit Posters Hansol Lim North Carolina State University, Wookhee Min North Carolina State University, Jessica Vandenberg North Carolina State University, Veronica Catete North Carolina State University, Judith Uchidiuno Georgia Institute of Technology, Bradford Mott North Carolina State University
	10:00 2h Talk		Assessing the Efficacy of Goal-Based Scenarios in Scaling AI Literacy for Non-Technical Learners Posters Ying Jui Tseng Carnegie Mellon University, Ruiwei Xiao Carnegie Mellon University, Christopher Bogart Carnegie Mellon University, Jaromir Savelka Carnegie Mellon University, Majd Sakr Carnegie Mellon University
	10:00 2h Talk		Linguistic Pedagogical Approaches to Transfer in Computer Science Posters Rosalind Owen WestEd
	10:00 2h Talk		Computer Science Kickstart: An Innovative Bootcamp to Ignite Passion in First-Year Female-Identifying University Students Posters Stella Kaval University of California, Berkeley, Mallika Reddy University of California, Berkeley, Abigail O'Neill University of California, Berkeley
	10:00 2h Talk		An investigation on Task Difficulty: Does Task Difficulty Depend on the Technology Used in Task Completion? Posters Mahir Akgun Pennsylvania State University, Sacip Toker Atilim University
	10:00 2h Talk		Scaling Collaborative Learning: Using the Community Digital Library to Enrich Course Content Posters Kevin Ros University of Illinois at Urbana-Champaign, ChengXiang Zhai University of Illinois at Urbana-Champaign
	10:00 2h Talk		A Cross-Case Analysis of Experienced Educators in CS Inclusion Posters Wei Yan University of Florida, Andrew B. Bennett University of Florida, Alexis Cobo University of Florida, Maya Israel University of Florida
	10:00 2h Talk		Student (Self) Assessment, Feedback, and Relatability in CS-2 Game Programming Assignments Posters Veena Ravishankar University of Mary Washington, Jessica Zeitz University of Mary Washington
	10:00 2h Talk		Root the (Ballot) Box: Designing Security Engineering Courses with E-Voting Posters Tushar Jois City College of New York, Atheer Almogbil Johns Hopkins University, Logan Kostick Johns Hopkins University
	10:00 2h Talk		Evaluating GPT-4 Code Generation as a Grading Mechanism for “Explain-in-Plain-English” Questions Posters David Smith University of Illinois at Urbana-Champaign, Craig Zilles University of Illinois at Urbana-Champaign
	10:00 2h Talk		Co-Designing Integrated CS Curriculum Artifacts with K-5 Classroom Teachers Posters Maverick Berner Marquette University, Maximus Berner St. Norbert College, Darcy Ronan Sacred Heart University, Dennis Brylow Marquette University
	10:00 2h Talk		A Study of Undergraduate Learning Assistants (ULAs) in Computer Science Posters Shirin Haji Amin Shirazi University of California, Riverside, Mariam Salloum Univeristy of California Riverside, Neftali Watkinson University of California, Riverside
	10:00 2h Talk		Generative AI as a Resource for Creativity in Computational Physics Posters Patti Hamerski Oregon State University
	10:00 2h Talk		Evaluating Storytelling Videos Using YouTube AnalyticsGlobal Posters Anna Ly University of Toronto Mississauga, Tingting Zhu University of Toronto Mississauga, Andrew Petersen University of Toronto
	10:00 2h Talk		See Logic In Life: An Individualized Exercise Structure for Making and Sharing Computing Concept Connections Posters Madalene Spezialetti Trinity College
	10:00 2h Talk		Experience Helps, but It Isn’t Everything: Exploring Causes of Affective State in Novice Programmers Posters Heidi Reichert North Carolina State University, Sandeep Sthapit North Carolina State University, Benyamin Tabarsi North Carolina State University, Ally Limke North Carolina State University, Thomas Price North Carolina State University, Tiffany Barnes North Carolina State University