Extracting and Transferring Hierarchical Knowledge to Robots Using Virtual Reality

We study the knowledge transfer problem by training a task of folding clothes in the virtual world using an Oculus Headset and validating with a physical Baxter robot. We argue such complex transfer is realizable if an abstract graph-based knowledge representation is adopted to facilitate the process. An And-Or-Graph (AOG) grammar model is introduced to represent the knowledge, which can be learned from the human demonstrations performed in the Virtual Reality (VR), followed by the case analysis of folding clothes represented and learned by the AOG grammar model. In the experiment, the learned knowledge from the given six virtual scenarios is implemented on a physical robot platform, demonstrating that the grammar-based knowledge is an effective representation.