With the development of science and technology, many precision fields have higher requirements for the machining quality and precision of hard and brittle materials. Brittle materials such as glass and ceramics have fragile physical properties, resulting in poor stability during machining. Ultrasonic vibration-assisted polishing (UVAP) is suitable for machining various brittle materials and can well solve the machining problems caused by the characteristics of brittle materials. In order to ensure the machining quality, a polishing force model of K9 optical glass for different processing parameters is established in this paper. The new model mainly considers the following aspects: (1) Considering the randomness of shape, the truncated polyhedral model is used to model the abrasive particles. (2) Based on the fractal theory and mathematical statistics, a microscopic morphology model of the polishing tool is established. (3) Based on the N–S equation, a polishing force model considering the micro-contact states of the polishing tool, abrasive particles, and workpiece is established. By comparing with the experimental results, the new model has high accuracy in predicting the polishing force. The new model and results lay the foundation for the subsequent research of brittle materials.