Proposal-Based Graph Attention Networks for Workflow Detection

In the process of “Industry 4.0”, video analysis plays a vital role in a variety of industrial applications. Video-based action detection has obtained promising performance in computer vision community. However, in complex factory environment, how to detect workflow of both machines and workers in production process is not well resolved. To solve this issue, we propose a generic proposal based Graph Attention Networks for workflow detection. Specifically, an efficient and effective action proposal method is firstly employed to generate workflow proposals. Then, these proposals and their relations are exploited for proposal graph construction. Here, two types of relationships are considered for identifying the workflow phases, which are contextual and surrounding relations to capture context information and characterize the correlations between different workflow instances. To improve the recognition accuracy, within-category and between-category attention are incorporated to learn long-range and dynamic dependencies respectively. Thus, the capability of feature representation for workflow detection can be greatly enhanced. Experimental results verify that the proposed approach is considerably improved upon the state-of-the-arts on THUMOS’14 and a practical workflow dataset, achieving 6.7% and 3.9% absolute improvement compared to the advanced GTAN detector at tIoU threshold 0.4, respectively. Moreover, augmentation experiments are carried out on ActivityNet1.3 to prove the effectiveness of performance improvement by modeling workflow proposal relationships.