How to Evaluate Mobile 360° Video Streaming Systems?

360 degrees video brings an immersive experience to users by projecting panoramic content onto the display. Viewing 360 degrees videos on untethered mobile platforms further enhances the immersive user experience. As a result, a large number of systems for optimized high-resolution 360 degrees video streaming to mobile devices have been proposed over the past few years. In this paper, we review the diverse set of research methodologies in the system design and evaluation of recently proposed mobile 360 degrees video streaming systems and discuss a number of pitfalls that prevent a fair and meaningful comparison among different systems. Our discussion suggests that there is an urgent need to redefine the design objectives and to develop an effective methodology for a meaningful evaluation and comparison of different systems. We finish with a set of concrete guidelines on the design and evaluation methodology of future mobile 360 degrees video systems.