Transform Domain Temporal Prediction and Geodesic Motion Compensation in Spherical Video Coding

This paper considers spherical videos dominated by camera motion, which are prevalent in many virtual and augmented reality applications whose deployment critically depends on efficient compression that is tailored to the signal characteristics. Existing approaches project the spherical video onto a plane (or planes) via one of several known projection geometries, followed by compression by a standard 2D codec. However, performance is compromised by the fact that motion in the projected domain is warped by the projection, and is not effectively compensated with the simple motion model employed. This paper leverages our recently proposed geodesic translation motion model to capture the exact pixel-wise motion on the sphere, and embeds it within a transform-domain temporal prediction framework, which is designed to disentangle spatial and temporal correlations. The approach circumvents a major shortcoming of standard temporal prediction, which effectively consists of simple pixel-copying (after motion compensation) from the reference frame, and thus largely ignores underlying spatial correlations. Transform-domain temporal prediction exploits the (spatial) decorrelating properties of the transform, accounts for both spatial and temporal correlations, and explicitly captures variations in temporal correlations across frequencies. We design the correlation filters by using an iterative open loop procedure that asymptotically converges to closed loop operation. Experimental results provide evidence that the overall approach, comprising geodesic motion compensation in conjunction with transform domain temporal prediction, offers considerable gains over the state-of-the-art.