Network synthesis through data-driven growth and decay

An algorithm for addition and deletion (ADDEL) of resources during learning is developed to achieve two goals: (1) to find feed-forward multilayer networks that are as small as possible, (2) to find an appropriate structure for such small networks. These goals are accomplished by operating alternately between an adding phase and a deleting phase while learning the given input-output associations. The adding phase develops a crude structure by filling in resources (connections, units and layers) at a virtual multilayer network with a maximum of L possible layers. The deleting phase then removes any unnecessary connections to obtain a refined structure. The additions and deletions are done based on a sensitivity measure and a corresponding probability rule so that only the synapses which are most effective in reducing the output error are preserved. A generalization error estimated from a validation set is used to control the alternation between the two learning phases and the termination of learning. Simulations, including hand-written digit recognition, demonstrate that the algorithm is effective in finding an appropriate network structure for a small network which can generalize well. The algorithm is used to investigate when the size of a network is important for generalization. (C) 1997 Elsevier Science Ltd.