Balancing exploitation and exploration in parallel Bayesian optimization under computing resource constraint

Parameter survey with an MPI application is extensively used, where the optimality of each parameter setting is evaluated by executing the application, called a trial. Bayesian optimization can find a suboptimal parameter setting with less trials. Parallel Bayesian optimization (PBO) is to execute multiple trials in parallel to reduce the execution time. However, evaluating too many trials at once is likely to be trapped by a local minimum, and degrade the final solution. In addition, under a computing resource constraint, the more parallel trials are executed, the less computing resource each MPI application run can use. As a result, the execution time might become longer. Thus, there is a trade off in the number of parallel trials, and the parameter search generally becomes less explorative and more exploitative by decreasing the number of parallel trials. This paper proposes a method to dynamically adjust the number of parallel trials accordingly to the degree of progress in optimization. The evaluation results demonstrate that the proposed method can adapt to the optimization problems and hence is robust to the problem. As a result, the proposed method can consistently achieve both a faster improvement speed and a better final solution in many cases.