The dynamics of on-line principal component analysis

The learning dynamics of an on-line algorithm for principal component analysis is described exactly in the thermodynamic limit by means of coupled ordinary differential equations for a set of order parameters. It is demonstrated that learning is delayed significantly because existing symmetries among student vectors have to be broken. A closely related effect is the perfect or partial loss of initial knowledge in the course of learning. The analysis shows that different learning rates for the student vectors improve the performance of the algorithm drastically.