Genetic Algorithm (GA) has been successfully applied to codebook design for vector quantization and its candidate solutions are normally turned by LBG algorithm. In this paper, to solve premature phenomenon and falling into local optimum of GA, a new Genetic Simulated Annealing-based Kernel Vector Quantization (GSAKVQ) is proposed from a different point of view. The simulated annealing (SA) method proposed in this paper can approach the optimal solution faster than the other candidate approaches. In the frame of GA, firstly, a new special crossover operator and a mutation operator are designed for the partition-based code scheme, and then a SA operation is introduced to enlarge the exploration of the proposed algorithm, finally, the Kernel function-based fitness is introduced into GA in order to cluster those datasets with complex distribution. The proposed method has been extensively compared with other algorithms on 17 datasets clustering and four image compression problems. The experimental results show that the algorithm can achieve its superiority in terms of clustering correct rate and peak signal-to-noise ratio (PSNR), and the robustness of algorithm is also very good. In addition, we took "Lena" as an example and added Gaussian noise into the original image then adopted the proposed algorithm to compress the image with noise. Compared to the original image with noise, the reconstructed image is more distinct, and with the parameter value increasing, the value of PSNR decreases.