Recently, the potential of microalgae in wastewater treatment has attracted attention. The goal of this study is to find optimum conditions for microalgae growth and the concentration of cheese whey wastewater (CWW) to get the best treatment efficiency by using response surface methodology (RSM) and the decision tree algorithm for different pollutant parameters. The study used reactors with different amounts of CWW and Nannochloropsis sp. to find the best concentrations for each parameter. The best concentration of CWW was found to be 8000 mgCOD/L, and the best concentration of Nannochloropsis sp. microalgae was found to be 2200 mgVS/L. It was found that Chemical Oxygen Demand (COD), Total Organic Carbon (TOC), Total Kjeldahl Nitrogen (TKN), and Orthophosphorus (Ortho-P) could be removed at different ranges, 77-96 %, 95-98 %, 51-97 %, and 60-99 % of CWW, respectively, depending on the different combinations of microalgae and CWW concentrations. The desirability values in RSM for COD, TOC, TKN, and Ortho-P parameters to be 0.99, 0.94, 0.78, and 0.63, respectively. The study suggests the marine microalgae (Nannochloropsis sp.) could be an alternative way to treat saline CWW, to create a circular economy. The machine learning (ML) method validates that RSM predictions are consistent and accurate. The results show that it is possible to combine traditional optimization methods with more advanced ML methods to facilitate the design and the operation of the treatment plants.