Impact of assignment completion assisted by Large Language Model-based chatbot on middle school students' learning

With the prevalence of Large Language Model-based chatbots, middle school students are increasingly likely to engage with these tools to complete their assignments, raising concerns about its potential to harm students' learning motivation and learning outcomes. However, we know little about its real impact. Through quasi-experiment research with 127 Chinese middle school students, we examined the impact of completing assignments with a Large Language Model-based chatbot, wisdomBot, on middle school students' assignment performance, learning outcomes, learning motivation, learning satisfaction, and learning experiences; we also summarized teachers' reflections on learning design. Compared to control groups, the Large Language Model chatbot-assisted group demonstrated significantly higher assignment submission rates, word counts, and scores in assignment performance. However, they gained significantly lower scores on materials refinement and knowledge tests. No significant differences have been observed in learning motivation, satisfaction, enjoyment, and students' ability to migrate their knowledge. The majority of students expressed satisfaction and willingness to continue using the tool. We also identified three key gaps in learning designs, including providing scaffolds for the potential prompts, suggesting group collaboration mode, and relinquishing the authoritarian of the teacher. Our findings provide insights regarding with Large Language Model-based chatbots we could better design assignment assessment tools, facilitate students' autonomous learning, provide emotional support, propose guidelines and instructions about applying Large Language Model-based chatbots in K-12, as well as design specialized educational Large Language Model-based chatbots.