Background Oral squamous cell carcinoma (OSCC) is a common skin/oral malignant tumor. In recent years, advancements in the field of medical technology resulted in significant improvement in the treatment and management of OSCC. However, adverse reactions associated with the application of radiotherapy and chemotherapeutic drugs resulted in a low overall survival rate. Therefore, it is important to develop and explore alternative chemotherapy drugs, which can effectively treat OSCC, and clarify their mechanism of action. Objective The present study aimed to investigate the effect of three hexa-coordinated octahedral nickel (II) complexes, namely [Ni(sal-L-phe)(phen)(CH3OH)]center dot CH3OH (1), [Ni(naph-L-phe)(phen)(CH3OH)] (2), and [Ni(o-van-L-phe)(phen) (CH3OH)[center dot 5CH(3)OH (3), on proliferation, migration, and apoptosis of CAL-27 cells. Results The results of the study showed that complex 3 exhibited sensitive/significant cytotoxicity toward CAL-27 cells, with an IC50 value of 15.90 mM. Further, an assessment of the underlying mechanism showed that complex 3 could significantly increase autophagy and reactive oxygen species (ROS) levels in CAL-27 cells. At the same time, it reduced mitochondrial membrane potential (MMP) in a dose-dependent manner. Additionally, complex 3 might act via inhibition of proliferation and migration of CAL-27 cells and induction of apoptosis, which was mediated via regulation of the expression of Bcl-2 family proteins in vitro. Conclusion The results of the study showed that nickel complexes could potently inhibit the growth of CAL-27 cells, which was mediated via mitochondrial dysfunction, intracellular ROS accumulation, and ROS-mediated DNA damage. Altogether, the findings of the study would provide new leads for the designing and synthesis of novel metal drugs for OSCC.