Uncovering Behavioral Patterns in Student-LLM Conversations during Code Refactoring Tasks

Abstract

The ability to write high-quality code is essential in software development. With the growing use of LLMs for code generation, ensuring code quality has become even more important, as AI-generated code often contains various quality issues. Educators are increasingly integrating LLMs into programming education, which has raised interest in understanding how students interact with LLMs. To date, little research has explored the teaching and learning of refactoring supported by LLMs. However, before investigating whether and how to integrate LLMs to support students with refactoring, it is important to understand how students interact with these models. In this study, we analyze students' behavior when interacting with an LLM during code refactoring tasks. We identify six patterns of student-LLM interaction behavior, and explore the relation between these patterns and the quality of student code. The two most common behaviors are: (1) students who delegate the entire refactoring task to the LLM; and (2) students who only use the LLM to ask questions about their refactored code. We observed a relation between the latter behavior and solutions of good quality.