Patterns of genital arousal to sexual stimuli are somewhat different between men and women. Heterosexual males and homosexual males show clear category specific arousal that is consistent with their self-reported sexual preference. However, heterosexual women do not show this category specificity. In the present study, we attempted to measure a person’s automatic appraisals of stimuli with respect to the concept of sex via the use of implicit measures (the Implicit Association Test and the priming task). In three experiments, we showed that heterosexual females did not show a sex-related category specific response in favor of male versus female stimuli. However, this lack of specificity was not due to a lack of sex-related appraisals, but by equal appraisals of both male and female stimuli. On the other hand, heterosexual men, homosexual men, and homosexual women all showed automatic sex-related appraisals of stimuli that were category specific and in line with their self-reported sexual preference. The study shows difference in the pattern of sexual interest between genders at the earliest stages of the evaluation of a stimulus.