l1 Regularization in Two-Layer Neural Networks

A crucial problem of neural networks is to select an architecture that strikes appropriate tradeoffs between underfitting and overfitting. This work shows that l(1) regularizations for two-layer neural networks can control the generalization error and sparsify the input dimension. In particular, with an appropriate l(1) regularization on the output layer, the network can produce a tight statistical risk. Moreover, an appropriate l(1) regularization on the input layer leads to a risk hound that does not involve the input data dimension. The results also indicate that training a wide neural network with a suitable regularization provides an alternative bias-variance tradeoff to selecting from a candidate set of neural networks. Our analysis is based on a new integration of dimension-based and norm-based complexity analysis to bound the generalization error.