Efficient training of Time Delay Neural Networks for sequential patterns

Time Delay Neural Networks are an extension of the classical multi-layer perceptron with time-delayed links. They are used to deal with sequence recognition problems in which a finite memory of past events is sufficient. Usually Time Delay Neural Networks are trained by performing a complete spatial expansion of delayed links through time to reconduct the training to that of a feedforward network. But this complete expansion is unnecessary. In fact it is sufficient to combine a partial spatial expansion with a sliding input window to obtain the same result. In this way we exploit the computational efficiency of standard backpropagation while increasing the flexibility of the method to deal with variable length sequences and reducing the storage occupation. In this paper a general training algorithm for Time Delay Neural Networks is presented, showing in detail the formal differences with respects to error backpropagation for feedforward networks. Furthermore, an efficient implementation is described, which exploits a partial spatial expansion of delayed links, showing formally its equivalence with the general algorithm.