Conditional variance estimation using stochastic learning algorithm

Aritificial neural networks may be used for a function approximator which includes not only deterministic but also probabilistic model. Conditional variance estimation using a neural network is a good example of probabilistic model approximation, because conditional variance, which is a function of input variable, is an important parameter to describe a Gaussian probabilistic model. The majority of learning algorithms are based on a concept of likelihood maximization or expectation maximization method. This article presents an alternative learning algorithm based on a different concept for a multilayer perceptron. The proposed variance learning algorithm can be regarded as a kind of modified delta rule, where delta is determined by an iterative estimation algorithm, which is also proposed in this article. The proposed learning algorithm has stochastic property because the delta is stochastically determined by the estimation algorithm. Relationships of delta to the transient and steady state of the learning process are also stochastic. First, the iterative variance estimation algorithm is explained. Second, the transient state behavior is investigated to have an insight into convergence and stability properties with respect to delta. Third, the steady state analysis is described to show the relationship of delta to steady state error bound. Theoretical analysis on steady state behavior produces analytic formula for steady state error bound of the variance learning algorithm in terms of the delta. Finally, multilayer perceptron using the proposed learning algorithm is simulated for the demonstration of variance estimation.