Multimodal Effects of Color and Haptics on Intuitive Interaction with Tangible User Interfaces

Tangible User Interfaces (TUIs) allow users to sense and manipulate digital information through physical objects. Although haptic properties are emphasized, TUIs are presented in and perceived through multiple modalities. Especially visual properties like color shape the users' expectations about the relation between tangibles and the abstract data they represent and control. Viewing TUIs as multisensory percepts, we present an empirical study that quantifies benefits of an explicit design for color for intuitive interaction. In a cross-cultural experiment, 75 participants (Germans and Japanese) matched tangible objects of different colors, sizes, weights or temperatures with abstract words. The results indicate that multimodal representations increase the efficiency, effectiveness and user satisfaction, but only if colors and haptic properties evoke congruent associations. Incongruently designed tangibles led to a 17% increase in response speed, -28% lower accuracy and -61% lower preference ratings compared to congruently designed tangibles.