Neural network hyperparameter optimization based on improved particle swarm optimization

Hyperparameter optimization is considered as one of the most challenges in deep learning and dominates the precision of model in a certain. Recent proposals tried to solve this issue through the particle swarm optimization(PSO), but its native defect may result in the local optima trapped and convergence difficulty. In this paper, the genetic operations are introduced to the PSO, which makes the best hyperparameter combination scheme for specific network architecture be located easier. Specifically, to prevent the troubles caused by the different data types and value scopes, a mixed coding method is used to ensure the effectiveness of particles. Moreover, the crossover and mutation operations are added to the process of particles updating, to increase the diversity of particles and avoid local optima in searching. Verified with three benchmark datasets, MNIST, Fashion-MNIST, and CIFAR10, it is demonstrated that the proposed scheme can achieve accuracies of 99.58%,93.39%, and 78.96%, respectively, improving the accuracy by about 0.1%, 0.5%, and 2%,respectively, compared with that of the PSO.