Recognizing Text Using Motion Data From a Smartwatch

In this paper, we show how motion data collected with a smartwatch can be used to infer text written on a whiteboard. All data is gathered by a commercial, off the shelf smartwatch, without any additional hardware. We obtain single letters recognition rates of 94% in the first guess. Using three guesses, we recognize 99% of the letters. This result is achieved by transforming raw sensor measurements into a coordinate system reflecting the orientation of the whiteboard. Therefore, training data recorded on one whiteboard can be used on another even if the user is writing at different heights or body postures. We show that the built in microphone picks up the sounds caused by the pen which can help to segment the input into individual letters.