Toward Programming a Collaborative Robot by Interacting with Its Digital Twin in a Mixed Reality Environment

In an industrial production context, when a robotic arm is assigned a new task, it must be re-programmed by specialized personnel with the necessary skills and knowledge. In addition, the robot remains offline while being re-programmed, negatively affecting productivity. Mixed reality opens up the opportunity for a production worker with no programming skills to teach a robot by interacting with the hologram of its digital twin, thus not interfering with the physical robot during re-programming. We describe the design and implementation of a mixed-reality interface to instruct trajectories for a robot end-effector by moving the hologram of its digital twin by hand. The interface supports the integrality of this interaction so that the user can configure the end effector location and orientation simultaneously. We also report a user study (n = 14) to characterize this interface in terms of usability, user experience, and estimated temporal efficiency that it can provide when programming robot trajectories.