Adaptive structure feed-forward neural networks using polynomial activation functions

In cascade-correlation (CC) and constructive one-hidden-layer networks, structural level adaptation is achieved by incorporating new hidden units with identical activation functions one at a time into the active evolutionary net. Functional level adaptation has not received considerable attention, since selecting the activation functions will increase the search space considerably, and a systematic and a rigorous algorithm for accomplishing the search will be required as well. In this paper, we present a new strategy that is applicable to both the fixed structure as well as the constructive network trainings by using different activation functions having hierarchical degrees of nonlinearities, as the constructive learning of a one-hidden-layer feed-forward neural network (FNN) is progressing. Specifically, the orthonormal Hermite polynomials are used as the activation functions of the hidden units, which have certain interesting properties that are beneficial in network training. Simulation results for several noisy regression problems have revealed that our scheme can produce FNNs that generalize much better than one-hidden-layer constructive FNNs with identical sigmoidal activation functions, in particular as applied to rather complicated problems.