Creating the Future: Augmented Reality, the next Human-Machine Interface

XR, consisting of Virtual Reality (VR) and Augmented Reality (AR) together, will be the next general computing platform, dominating our relationship with the digital world for the next 50 years, much as personal computing has dominated the last 50. XR will be the way people work, play, and connect. VR headsets will create deeply immersive new experiences and will enable the richest Metaverse experiences. Always-available AR glasses will be with us when we're on the go, letting us act with extremely low latency and friction, allowing us to use virtual entities of all sorts to annotate our world, share with others, and communicate, and extending our perceptions, memory, and cognition with an Artificial Intelligence (AI) assistant that truly understands our context and our personal needs. Both will be centered around people rather than technology, allowing us to connect more strongly than ever before, and providing a contextually personalized interface that is far more intuitive and natural than anything that exists today. However, VR and particularly AR are at a very early stage, and need a great deal of innovation and development, across multiple technologies, before they can offer the level of performance that would cause billions of people to make XR a part of their everyday lives. Those technologies include optics, projectors, display systems, graphics, audio, hand tracking, eye tracking, face tracking, body tracking, world mapping and reconstruction, contextual understanding, sensors, interaction, and AI. VR headsets operate within tight power, weight, and thermal budgets, and AR glasses will need to operate within extraordinarily tight budgets; for AR glasses, the constraints include light weight, very low power, and all-day comfort and all-day operation in a socially acceptable form factor,. Breakthroughs are needed in every one of the areas listed above, and in almost every case, innovative, highly customized semiconductor technologies will be needed to achieve those breakthroughs. This will notably require new materials, miniaturization of the key components, and very-large-scale 3D heterogeneous integration of circuits and systems to create new functions. In addition, specialization at the level of architecture and domain-oriented accelerators, especially with respect to Machine Learning (ML), will be essential, and algorithm optimization will need to evolve in new directions. Finally, the challenges of AR can only be solved by true co-design of hardware and software; technology development will need to be driven from end to end, based on delivering important use cases with a full-system approach. This will create unforeseen opportunities and challenges for semiconductor technologies for decades to come.