An intelligent playback control system adapted by body movements and facial expressions recognized by OpenPose and CNN

Users watch videos for a long time when they are learning or entertaining. They may inevitably be tired, doze off or leave temporarily. However, the videos continue to play. When the users want to watch the video again, they must return to find the appropriate restarting position, causing inconvenience. The ultimate need of this study is to implement an effective video playback control system to automatically pause video playback when a user leaves the seat or falls asleep, while the system continues to play videos when the user sits in front of the computer and is in good condition. The proposed system recognizes human body movements and the opening/closing of the eyes (OCE). First, the user's image is captured through a web camera. Then the OpenPose deep-learning neural network recognizes the human pose. The recognized results are used to determine whether the user leaves or lies on her/his stomach. Therefore, the video can be paused if the user falls asleep while sitting with his eyes closed. The novelty of this study is that the proposed playback control system automatically pauses the video when the user is not in good condition. Accordingly, the user is free from wasting time searching for a proper playback position when the user wants to continue watching the video. The experimental results show that the accuracy rates of body motion recognition can reach 99.5%, and the accuracy rate of eyes closed recognition can reach 99.58%. Consequently, the proposed system can effectively control video playback in practice.