The Future of Interaction with Mobile Game Characters

This poster showcases the future of player-NPC interaction in gaming: more natural, based on speech. The poster explains the implementation of verbal interaction with an NPC in a mobile game. Large Language Models (LLMs) open new ways of interacting in gaming, but the large size and memory footprint make deploying them on mobile a challenge. We have identified a 33M-parameter small Language Model (LM), TinyStories [Eldan and Li 2023], re-purposed it to be conversational, and combined it with the standard Unity Multi-Layer Perceptron (MLP) ML-Agents [Juliani et al. 2020] model. The LM drives the interaction with the user, while the MLP performs the actions. A Sentence Similarity model [Reimers and Gurevych 2019] is then responsible for filtering user input to decide whether the NPC needs to communicate with the player or to perform an action. Running all these models locally on mobile inside a game engine like Unity can be very challenging in terms of optimization, but this poster demonstrates how it is possible.