Non-normality in Neural Networks

A central goal of neuroscience is to link synaptic connectivity of neural circuits to produced dynamics and computations. Anatomical and functional connectivity within neural systems is asymmetric, which upon linearization gives rise to non-normal dynamics. Particular linear combinations of neurons that are involved in circuit function are canonically identified in systems neuroscience via PCA, which seeks subspaces which maximize variance. We have recently proposed Dynamical Components Analysis (DCA), which seeks subspaces of activity in which the mutual information between past and future activity (i.e., 'the dynamic memory') is largest. Here, we show that the presence of non-normality leads to a divergence between these subspaces and consequently, the importance of single neurons that are identified by each method. Applied to in-vivo electrophysiology recordings from diverse brain areas, subspaces of past-future mutual information are better able to predict animal and human behavior than subspaces of high variance. Finally, we discuss possible consequences of non-normality for the training and function of in-silico recurrent neural networks.