MASSIVELY PARALLEL ARCHITECTURES FOR LARGE-SCALE NEURAL NETWORK SIMULATIONS

A toroidal lattice architecture (TLA) and a planar lattice architecture (PIA) as massively parallel architectures of neurocomputers for large scale neural network simulations are proposed. The performances of these architectures are almost proportional to the number of node processors and they adopt the most efficient two-dimensional processor connections to be implemented by the wafer scale integration (WSI) technology to date. They also give a solution to the connectivity problem, the performance degradation caused by the data transmission bottleneck, and the load balancing problem for efficient parallel processing in large scale neural network simulations. Furthermore, these architectures will have an even greater expandability of parallelism and exhibit great flexibility for the various configurations of neural networks and varieties of neuron models. First we define the general neuron model that is the basis of these massively parallel architectures. Then, we take a multilayer perceptron (MLP) as a typical example of neural networks and describe the simulation of the MLP using error back propagation learning algorithms on virtual processors (VP's) with the TLA and the PLA. Then, the mapping from the VP's to physical node processors with the same TLA and PLA is presented. This mapping is done by row and column partitions. At the same time, the row and column permutations are carried out for node processor load balancing. The mapping algorithm for the load balancing is given. An equation to estimate the performance of these architectures is also presented. Finally, we describe implementation of the TLA with transputers including a parallel processor configuration, load balance algorithm, and evaluation of its performance. We have implemented a Hopfield neural network and a MLP, and applied them to the traveling salesman problem (TSP) and the identity mapping (IM), respectively. The TLA neurocomputer has achieved 2 MCPS in a feedforward network and 600 KCUPS in a back propagation network using 16 transputers. Actual proof that its performance increases almost in proportion to the number of node processors is given.