Purpose: The geometrical uncertainties in the patient positioning during intensity modulated radiotherapy (IMRT) are crucial as there is potential to underdose the tumor and overdose the nearby critical structures. Image guided techniques provide a solution to assess the patient set-up uncertainties and help determine the optimal planning target volume (PTV) margin to the clinical tumor volume (CTV). Methods: A retrospective study was conducted to evaluate patient set-up errors along the three translational directions at different treatment sites such as the brain, the head and neck (H&N) and the prostate. A total of 60 patients' set-up error data was analysed to evaluate the systematic and random errors and the optimal CTV-PTV margin. Results: For brain and H&N sites, more than 90, 80 and about 100% of the total image acquisitions were less than 3 mm in lateral, longitudinal and vertical directions respectively. For the prostate cases, the frequency of patient set-up error to be less than 3 mm were 79.7, 75.6 and 80% in lateral, longitudinal and vertical directions respectively. About 0.6% had more than 7 mm error in the lateral and longitudinal directions for the prostate site. CTV-PTV margin of 3.4, 3.4 and 1.9 mm for brain cases, 3.5, 3 and 1.8 mm for H&N cases and 5, 4.6 and 4.5 mm for the prostate cases in the lateral, longitudinal and vertical directions respectively were determined. Conclusion: Image guidance is an effective method to evaluate the accuracy of IMRT treatment delivery. The optimal CTV-PTV margin can be determined to ensure adequate dose to CTV, specific to the site.