Object tracking and object change detection in desktop manipulation for video-based interactive manuals

This paper introduces a novel method for object tracking and recognition in assembly work. The purpose of this study is to index instructional videos and to provide appropriate instructions to a user during actual assembly work. Object tracking for this purpose involves a lack of prior knowledge such as an object's shape or color, since objects are often moved, assembled, or even crushed. The clutter present in an environment or environmental changes must also be addressed. For this purpose, we use two or more pairs of image sensors. In this method, an object held by a hand is reliably detected, and its 3D area, that is, its volume and location, are obtained using shape-from-silhouette in real time. The observation of such volume allows the estimation of the changes in an object's state, and can be good indices for the processes of assembly work.