In five-axis machining, the setup of workpiece on the machine's table is an essential element of the kinematic chain that affects the performance of machine tools, such as the kinematic behavior and machining accessibility. In this paper, a mathematical model between singularity, a special kinematic behavior of five-axis machine tool, and the workpiece setup is built, and another model between accessibility of machine tool and the workpiece setup is presented. Regarding the above two models, a geometry-based approach is presented to unify and simplify them. By considering the two constraints of singularity-free and full accessibility, a general framework for the workpiece setup optimization of five-axis machining is established, from which the orientation and position of the workpiece on the table is optimized. To validate the advantage of the proposed framework, some physical cutting experiments and simulation are implemented toward a typical five-axis machine tool, testifying that the optimal setup can be achieved efficiently with the kinematic performance, surface machining quality as well as the machining efficiency significantly improved compared with the benchmark.