Mixed Reality Tailored to the Visually-Impaired

The goal of the proposed device and software architecture is to apply the functionality of Mixed Reality (MR) in order to make a virtual environment that is more accessible to the visually-impaired. We propose a glove-based system for MR that will use finger and hand movement tracking along with tactile feedback so that the visually-impaired can interact with and obtain a more detailed sense of virtual objects and potentially even virtual environments. The software architecture we propose is making the current MR frameworks more accessible by augmenting the existing software and extensive 3D model libraries with both the interfacing of the glove-based system and the audibly navigable user interface (UI) of a virtual environment we have developed. We have implemented a circuit with flex sensor flexion angle tracking of a single hand and variable vibration intensities for the vibromotors on all 5 fingertips of a single hand. Alongside the circuit, we have developed a virtual environment that can be hosted on a Windows 10 application. The virtual hand and its fingers can be moved with the system's different inputs and the virtual fingertips touching the virtual objects trigger vibration motors (vibromotors) to vibrate while the virtual objects are being touched. A rudimentary implementation of picking up and moving virtual objects inside the virtual environment has also been implemented. In addition to the vibromotor responses, text to speech (TTS) has also been implemented in the application for when virtual fingertips touch virtual objects and other relevant events in the virtual environment. A visual UI has also been designed and developed for the aiding of current and future development.