Estimating heading from optic flow: Comparing deep learning network and human performance

Convolutional neural networks (CNNs) have made significant advances over the past decade with visual recognition, matching or exceeding human performance on certain tasks. Visual recognition is subserved by the ventral stream of the visual system, which, remarkably, CNNs also effectively model. Inspired by this connection, we investigated the extent to which CNNs account for human heading perception, an important function of the complementary dorsal stream. Heading refers to the direction of movement during self-motion, which humans judge with high degrees of accuracy from the streaming pattern of motion on the eye known as optic flow. We examined the accuracy with which CNNs estimate heading from optic flow in a range of situations in which human heading perception has been well studied. These scenarios include heading estimation from sparse optic flow, in the presence of moving objects, and in the presence of rotation. We assessed performance under controlled conditions wherein self-motion was simulated through minimal or realistic scenes. We found that the CNN did not capture the accuracy of heading perception. The addition of recurrent processing to the network, however, closed the gap in performance with humans substantially in many situations. Our work highlights important self-motion scenarios in which recurrent processing supports heading estimation that approaches human-like accuracy. © 2022 The Author(s)