Spatiotemporal-Memory-Guided Machine Perception for Augmented Reality

Augmented Reality (AR) devices are equipped with advanced sensors such as panoramic cameras. They can see or hear what humans cannot. The superior sensing capabilities - together with machine perception models - enable promising AR-based human assistants in daily tasks. Human task performance progressively enhances with accumulated memory, such as with observed objects' location in working memory, and training in long-term memory. We transform this cognitive nature into real-time machine perception models tailored for human-assisting AR applications. Instead of running machine perception models on each consecutive frame, we leverage the egocentric AR cameras to reconstruct a "4D memory map", which mimics the human spatiotemporal memory.