Detecting human-object interactions in videos by modeling the trajectory of objects and human skeleton

This article focuses on the task of detecting human-object interactions (HOI) in videos, with the goal of identifying objects interacting with humans and predicting human-object interaction classes. Two frame-works are proposed which detect human-object interactions in videos by modeling the trajectory of objects and human skeleton. The first framework (knowledge-based spatial-temporal HOI) treats the entire scene to be a HOI graph made up of the human skeleton and objects. It has fewer parameters and a higher possibility for knowledge embedding. The second framework (hierarchical spatial-temporal HOI) constructs a HOI graph after obtaining the feature of the human skeleton and objects. It outperforms the competition in terms of performance and generalization. Experimental results in CAD-120 dataset and SYSU-HOI dataset show that the proposed frameworks are more advanced than the state-of-the-art methods, with smaller parameters and shorter inference time. Such results confirm that the proposed frameworks effectively reduce parameters and inference time while maintaining detection accuracy in HOI videos.(c) 2022 Published by Elsevier B.V.