Color image multilevel thresholding segmentation based on metaheuristic optimization algorithm and histogram has been widely used in many fields, but histogram neglects the relationships between the neighborhood pixels and any correlation among gray levels of the image when selecting the optimum threshold values. Therefore, in this paper, a novel and efficient color image multilevel thresholding method is proposed which uses an energy function to generate the energy curve of an image by considering spatial contextual information of the image. The presented color image segmentation technique based on energy curve takes the between class variance, Tsallis entropy and Kapur’s entropy as objective functions. In order to further enhance the segmentation performance, an improved firefly algorithm (IFA) is presented in this paper. The IFA algorithm based on energy function is used for color image multilevel thresholding problem and compared with modified firefly algorithm (MFA), cuckoo search (CS), grasshopper optimization algorithm (GOA), Harris hawks optimization (HHO), emperor penguin optimization (EPO) algorithms. The experimental results are presented in terms of optimal threshold value, optimal objective function values, peak signal to noise (PSNR), structural similarity index (SSIM), standard deviation of the objective values and statistical results. The experimental results show that the presented method outperforms the other algorithms and Kapur’s entropy based on energy curve is better than between class variance and Tsallis entropy for color image multilevel thresholding segmentation.