Excessive fluoride levels in drinking water poses serious global health issues. Decreased male fertility rates are reported from fluorosis endemic regions globally. This study aimed to examine potential impacts of chronic exposure to fluoride in drinking water on male reproductive parameters. The case–control study was based on recruits from a fluorosis endemic region, Anuradhapura, and a non-endemic region, Colombo, (n = 15 fertile with proven paternity and 15 subfertile, from each region) in Sri Lanka. Seminal fluid analysis (SFA) was performed according to WHO guidelines. Fluoride ion-selective electrodes quantified serum fluoride levels, while sandwich ELISAs assayed seminal IL-4 and IL-1β cytokines. SFA between the fertile groups did not differ significantly (p > 0.05). Compared to the non-endemic subfertile group (NESG), the endemic subfertile group (ESG) manifested highly significant higher serum fluoride concentration, percentage of morphologically normal spermatozoa (p < 0.001), and liquefaction time (p < 0.02), while the contrary was true of semen volume (p < 0.001). Significantly lower spermatozoa count (p < 0.02), motility (p < 0.001), and viability (p < 0.002) detected in the ESG compared to the NESG, was reiterated by significant negative correlation established between serum fluoride concentration and aforementioned three spermatozoa parameters in ESG (p < 0.05). Seminal plasma IL-4 and IL-1β levels did not significantly differ between ESG and NESG (p > 0.05). The source of drinking water significantly differed in ESG (67% well water) and NESG (60% pipe-borne water) (p < 0.02). These findings were indicative of serum fluoride concentration as an important contributory factor for male factor subfertility in a fluorosis endemic region in Sri Lanka.