Gesture-based Human-Robot Jazz Improvisation

We present Shimon, an interactive improvisational robotic marimba player, developed for research in Robotic Musicianship. The robot listens to a human musician and continuously adapts its improvisation and choreography, while playing simultaneously with the human.We discuss the robot's mechanism and motion-control,which uses physics simulation and animation principles to achieve both expressivity and safety. We then present a novel interactive improvisation system based on the notion of gestures for both musical and visual expression. The system also uses anticipatory beat-matched action to enable real-time synchronization with the human player. Our system was implemented on a full-length human-robot Jazzduet, displaying highly coordinated melodic and rhythmic human-robot joint improvisation. We have performed with the system infront of alive public audience.