Structured LSTM for Human-Object Interaction Detection and Anticipation

Understanding human activities is one of the important tasks in. computer vision. Although a lot of efforts have been made, recognizing complex human activities such as human-object interactions remains challenging. In general, human-object interactions can be considered as a temporal sequence with the transition in relationships of humans and objects over the time. Recently, many studies have shown sequential learning power of Long Short-Term Memory (ISM) for long-term temporal dependency problems. In this work, we focus on the problem of modeling spatio-temporal relationships between objects and humans with LSTM for detecting and anticipating human-object interactions in daily life. Instead of considering only how human pose and human-object relations change during human activity, we also take the impact of human activity on the state of objects and the relationships between objects into account for labeling human-object interaction. We evaluated our method on a challenging human-object interaction dataset consisting of 120 videos with different high-level activities, sub-activities and object affordances, The experimental results showed the significant improvements in both detecting and anticipating interaction activities.