Purpose The study aimed to investigate the geochemical baseline concentrations (GBCs), pollution status, ecological risk, human health risk, sources, and source-specific risk of potentially toxic elements (PTEs) in soils around the Han River Reservoir (HRR). Materials and methods Topsoil and subsoil samples collected from HRR were analyzed for PTEs (As, Cd, Cr, Cu, Hg, Ni, Pb, Zn). The cumulative frequency distribution (CFD) curves were established to estimate the GBCs. The geoaccumulation index (I-geo), modified potential ecological risk index (NIRI), and human health risk assessment (HHRA) were committed to identifying pollution and risk characteristics. Absolute principal component score-multi-liner regression (APCS-MLR) and positive matrix factorization (PMF) models were applied to determine the sources and to quantitatively identify source-specific risks. Results and discussion The GBCs in HRR were significantly different from the background values of Hubei. The mean concentrations of Cd, Cr, Cu, Pb, and Zn were 1.6, 1.0, 1.2, 1.1, and 1.1 times higher than the GBCs, respectively. Cd and Hg were the primary contributors to ecological risk. As and Cr were the major elements affecting health risks, while those for humans were within acceptable limits. The main sources were atmospheric deposition (As, Pb), agricultural activities (Cd), soil parent material (Cr, Cu, and Ni), industrial activities (Zn), and mixed sources consisting of soil parent material and agricultural activities (Hg). Among them, agricultural activities, soil parent material, and atmospheric deposition were the main factors affecting ecological and human health. Conclusions The accumulation and contamination of PTEs were influenced by a combination of anthropogenic and natural factors. As, Cd, Cr, and Hg should be given special attention due to their contamination risks.