Segmentation and Recognition of Eating Gestures from Wrist Motion using Deep Learning

This paper describes a novel approach of segmenting and classifying eating gestures from wrist motion using a deep learning neural network. It is inspired by the approach of fully-convolutional neural networks in the task of image segmentation. Our idea is to segment 1D gestures the same way 2D image regions are segmented, by treating each inertial measurement unit (IMU) datum like a pixel. The novelty of our approach lies in training a neural network to recognize data points that describe an eating gesture just like it would be trained to recognize pixels describing an image region. The data for this research is known as the Clemson Cafeteria Dataset. It was collected from 276 participants that ate an unscripted meal at the Harcombe Dining Hall at Clemson University. Each meal consisted of 1 - 4 courses, and 488 such recordings were used for the experiments described in this paper. Sensor readings consist of measurements taken by an accelerometer (x, y, z) and a gyroscope (yaw, pitch, roll). A total of 51,614 unique gestures associated with different activities commonly seen during a meal were identified by 18 trained raters. Our neural network classifier recognized an average of 79.7% of 'bite' and 84.7% of 'drink' gestures correctly per meal. Overall 77.7% of all gestures were recognized correctly on average per meal. This indicates that a deep learning model can successfully be used to segment eating gestures from a time series recording of IMU data using a technique similar to pixel segmentation within an image.