Rectal adenocarcinoma has been a major cause of death for patients with colorectal adenocarcinoma for decades. Local recurrence is common, but it is difficult to manage if the lesion is not detected early because of the limitations of radiotherapy and chemotherapy. Therefore, this study aimed to determine the risk factors of pelvic recurrence and promote early intervention. In this case series analysis, we reviewed the medical records of patients in our section of Taipei Tzu Chi Hospital who were diagnosed with rectal cancer from 2007 to 2016. We compared and analyzed the patients’ clinical characteristics, pathological stage, tumor location, recurrent type, and outcome. A total of 238 patients who were diagnosed and received surgical treatments within this period were included in this study. The median distance of the primary tumor from the anal verge was 8.63 cm. There were 59 patients with tumor recurrence and 15 with pelvic recurrence. Recurrence type was associated with tumor location, but it was not statistically significant (6.53 SD4.02 vs. 8.23 SD4.02 vs. 9.28 SD5.24 cm, pelvis vs. lung vs. liver, respectively). Only the initial carcinoembryonic antigen (CEA) level was significantly different; however, there was no difference in T and N stages. The initial CEA level can predict pelvic recurrence level, which is more obvious than pathological stage and tumor location, and this result can help predict oncological outcome. Further evaluations with more cases and using other cell tools are suggested.