Sampling matters: SGD smoothing through importance sampling

Many authors have suggested studying the loss landscape of Deep Neural Networks as a tool to understand their generalisation capabilities, the performance of optimisation algorithms as well as to tailor acceleration schemes. Nonetheless, a complete understanding of the entire learning process is still missing. This is mainly due to the complexity of the loss landscape, characterised by a rich structure with many local minima and saddle points. The purpose of this paper is to study the stochastic nature of Stochastic Gradient Descent and its link with the loss function as well as to propose a data sampling scheme which favours smoothing of the loss landscape to accelerate convergence speed. We validate our sampling scheme on AlexNet and ResNet applied to the CIFAR-10 dataset.