More Robots are Coming: Large Multimodal Models (ChatGPT) can Solve Visually Diverse Images of Parsons Problems

Large language models are reshaping computing education. Based on recent research, these models explain code better than students, answer multiple choice questions at or above the class average, and generate code that can pass automated tests in introductory courses. In response to these capabilities, instructors have quickly adjusted their courses and assessment methods to align with shifting learning goals and the increased risk of academic integrity issues. While some scholars have advocated for the integration of visual problems as a safeguard against the capabilities of language models, new multimodal models now have vision and language capabilities that may allow them to analyze and solve visual problems. In this paper, we compare the large multimodal model (LMMs) GPT-4V with Bard, an LLM that uses Google Lens for text recognition. We find that LMMs, which have learned both pixel features (from images) and text features (from prompts) in the same embedding space, performed substantially better than Bard which uses a piecemeal approach. With a specific focus on Parsons problems presented across diverse visual representations, our results show that GPT-4V solved 96.7% these visual problems, struggling minimally with a single Parsons problem. Conversely, Bard performed poorly by only solving 69.2% of problems, struggling with common issues like hallucinations and refusals. These findings suggest that merely transitioning to visual programming problems might not be a panacea to issues of academic integrity in the generative AI era.