An LLM-based Reflection Analysis Tool for Identifying and Addressing Challenging Topics
Traditional evaluation of students’ learning primarily relies on assessing learning outcomes through summative or formative assessment methods. In these approaches, the primary emphasis is on the students’ learning outcome, rather than the learning process. However, assessing the learning process is as important since it allows providing timely feedback which can directly impact students’ learning outcomes. One of the known approaches to getting information about the learning process is the use of formative reflection tools, which also help students in developing their meta-cognitive skills. One effective method for formative reflection is the Minute Paper technique which asks students two concise questions after each class session: what they have learned and what challenges they have encountered. While Minute Papers encourage brief responses, the analysis process can become time-consuming as the number of students and class sessions grows. To address this challenge, in this study, we propose a Large Language Model (LLM)-based reflection analysis tool designed to assess the challenging topics students encounter during each class session. This tool suggests additional learning modules for students to study based on the frequency of the challenging topics. To achieve this, the model utilizes a local repository of lecture materials to create query contexts, which are then input into the LLM as prompts. Students are given access to these recommended resources for further learning, and they are encouraged to provide feedback after completing these modules. These data-driven recommended learning resources serve as continuous content delivery channels to foster a deeper understanding of the subjects at hand.