An Accessible, Browser-Based Gestural Controller for Web Audio, MIDI, and Open Sound Control

Handmate is a browser-based hand gestural controller for Web Audio, MIDI, and Open Sound Control (OSC), using open-source pose-estimation technology from Google MediaPipe. The Web browser was chosen as the platform to execute all computational processes so that users could access the latest computer vision technology for music-making without requiring any specialized hardware, software, or technical expertise. By combining preexisting technologies in a browser environment, we enable accessible, interactive sound-making using hand movement. Three controllers are described: (1) Handmate Effects, which receives microphone or user sound input and provides a choice of audio effects that can be controlled by hand movement; (2) Handmate MIDI, which outputs MIDI notes or control values; and (3) Handmate OSC, which outputs OSC messages for users to access hand-movement data in other applications.