Multimodal Learning of Keypoint Predictive Models for Visual Object Manipulation

Humans have impressive generalization capabilities when it comes to manipulating objects and tools in completely novel environments. These capabilities are, at least partially, a result of humans having internal models of their bodies and any grasped object. How to learn such body representations for robots remains an open problem. In this work, we present a self-supervised learning approach that extends a robot's kinematic model for object manipulation from visual latent representations. Our framework comprises two components: First, we present our multimodal keypoint detector: A neural network autoencoder architecture that fuses proprioception and vision during learning to predict visual key points on an object; second, we show how we can learn an extension of the kinematic chain of the robot by regressing virtual joints from the visual keypoints predicted by our multimodal keypoint detector. Our evaluation shows that our approach learns to consistently predict visual keypoints on objects in the manipulator's hand and, thus, can easily facilitate learning an extended kinematic chain to include the object grasped in various configurations, from a few seconds of visual data. Finally, we show that this extended kinematic chain lends itself for object manipulation tasks such as placing a grasped object and present experiments in simulation and on hardware.