Hyper-parameter Optimization Using Continuation Algorithms

Hyper-parameter optimization is a common task in many application areas and a challenging optimization problem. In this paper, we introduce an approach to search for hyper-parameters based on continuation algorithms that can be coupled with existing hyper-parameter optimization methods. Our continuation approach can be seen as a heuristic to obtain lower fidelity surrogates of the fitness function. In our experiments, we conduct hyper-parameter optimization of neural networks trained using a benchmark set of forecasting regression problems, where generalization from unseen data is required. Our results show a small but statistically significant improvement in accuracy with respect to the state-of-the-art without negatively affecting the execution time.