The measurement of traditional workpiece geometry is based on the measurement results of points, lines, surfaces, balls, and other elements as objects. The best measurement area in the measurement space of the on-machine measurement system for different typical measurement objects is determined to achieve a high-precision, efficient measurement. This study is oriented to the circular surface measurement object, establishes the integrated error model and circular surface measurement error model of the on-machine measurement system, and analyzes the circular surface measurement error distribution law in the designated measurement space. Beetle antennae search particle swarm optimization algorithm (BAS-PSO) is proposed to solve the optimal measurement area for the established theoretical objective model of the optimal measurement area. The optimization effect is compared between BAS-PSO and the ACO, PSO, FOA, SA-PSO (simulated annealing particle swarm optimization), and IA-PSO (immune particle swarm optimization) algorithms. The experiment of determining the best measurement area of the on-machine measurement system based on circular surface measurement is conducted, simulated, and calculated with the circular surface measurement error model. The comparative analysis of the best measurement area is obtained by algorithm optimization. Experimental results show that the BAS-PSO algorithm is not caught in the local optimum, its convergence speed and stability are better than those of the five other algorithms, and it is more suitable for solving the best measurement area of the circular surface measurement. The circular surface error measurement results and simulation calculations overlap with the best measurement area obtained by the algorithm optimization. The space of the best area is 356.061 mm ≤ X ≤ 365.061 mm, −109.727 mm ≤ Y ≤ −100.727 mm, and −263 mm ≤ Z ≤ −253 mm. The measured maximum error minimum value of this area is 5.3 μm. The minimum value of the maximum error obtained by the simulation calculation of the circular surface measurement is 4.1 μm, and the minimum value of the maximum error obtained by the algorithm optimization is 8.8 μm. The validity and practicability of the proposed method for determining the best measurement area of the three-axis machine tool on-machine measurement system are verified.