Internal Covariate Shift Reduction in Encoder-Decoder Convolutional Neural Networks

Internal covariant shift in deep neural networks affects the learning and convergence speed in ConvNets. Batch normalization was recently proposed to reduce the distribution of each layer's input to accelerate the training process. It also reduces overfitting and eliminates the need for using dropout in the fully connected layers, or RELU activation. Batch normalization, in its essence, seeks stable distribution of activation values throughout training, and normalizes the inputs of nonlinear data. In order to determine the usefulness of batch normalization in neural networks that don't use fully connected layers we evaluated the performance of an encoder-decoder ConvNet with and without using batch normalization. We found that batch normalization increased the learning performance by 18% but also increased the training time in each epoch (iteration) by 26%. The code for this work and the datasets are provided in a github repository.