The effects of smoking on hearing loss within the context of atherosclerosis was assessed, and the statistical interaction of occupational noise evaluated. A cross-sectional study was conducted in 397 Japanese males working at a metal factory, aged 21–66 years, in a periodical health checkup. The following information was obtained: two smoking indices of smoking status and Brinkman index, occupational noise exposures and atherosclerotic risk factors (body mass index, blood pressure, serum cholesterol, hemoglobin A1c, atherosclerosis index). Hearing acuity was measured at 4 kHz using a pure-tone audiometer in a quiet room. Among the total subjects, 55 (13.9%) were identified as having hearing loss at 4 kHz, and 151 (38.0%) were currently exposed to occupational noise. When adjusted for age and occupational noise exposure, odds ratios (95% confidence intervals) of hearing loss were 3.16 (1.04, 9.62) for past smokers and 3.39 (1.05, 11.01) for heavy smokers (Brinkman index >750 cigarettes per day × number of years), compared with never-smokers. Statistical interaction of occupational noise exposure was insignificant with the association between smoking and hearing loss. When including atherosclerotic risk factors in a multiple model, there were no significant associations between hearing loss and either smoking or any other factors (i.e., occupational noise and atherosclerotic factors). Smoking was found to be associated with hearing loss beyond occupational noise exposure, and this association seemed to be masked by atherosclerotic factors, suggesting that the direction of the atherosclerotic effect on the relationship might need to be explored between smoking and hearing impairment.