Challenges in Modelling Cooking Task Execution for User Assistance

Executing a complex physical task according to an instruction or a checklist is typical for various fields, such as aviation or healthcare. It is possible that the person is inexperienced or under stress and therefore unable to promptly consult the instruction text for correct execution of the task. To address this problem different works propose the usage of automated assistive systems that guide the user through the task execution. This is also addressed by the Defense Advanced Research Projects Agency (DARPA) Perceptually-enabled Task Guidance (PTG) program, aiming to provide users with augmented reality goggle capable of tracking the state of the user during task execution and to display hints relevant for the task. In this paper we discuss one important part of this project, namely, the ability to track the state of the user and the environment in order to be able to assist them. We discuss our modelling approach for the development of a probabilistic state tracker, which uses sensor observations to identify the current state and actions of the user, as well as the goal they are pursuing. The paper provides useful insights into the modelling of user behaviour, the challenges associated with modelling multi-user behaviour and how to tackle them.