The Optimization Landscape for Fitting a Rank-2 Tensor with a Rank-1 Tensor

The ability to approximate a multivariate function/tensor as a sum of separable functions/tensors is quite useful. Unfortunately, optimization-based algorithms to do so are not robust and regularly exhibit unusual transient behavior. A variety of different algorithms have been proposed with different features, but the state of the art is still unsatisfactory. In this work we step back from the algorithms and study the optimization problem that these algorithms are trying to solve. We apply dynamical systems concepts to analyze the simplest nontrivial case, which is a rank-2 tensor approximated by a rank-1 tensor. We find nonhyperbolic minima and saddles, which would be difficult for algorithms to handle. These features occur at relatively small but nonzero angles, rather than in the small-angle limit as the literature would suggest. We also identify transverse stability as a mechanism that may explain the slow convergence of these algorithms more generally.