The purpose of this study is to investigate the interaction between processing lexical and speaker-specific information in spoken word recognition. The specific question is whether repetition and semantic/associative priming is reduced when the prime and target are produced by different speakers. In Experiment 1, the prime and target were repeated (e.g., queen–queen) or unrelated (e.g., bell–queen). In Experiment 2, the prime and target were semantically/associatively related (e.g., king–queen) or unrelated (e.g., bell–queen). In both experiments, the prime and target were either produced by the same male speaker or two different male speakers. Two interstimulus intervals between the prime and target were used to examine the time course of processing speaker information. The tasks for the participants included judging the lexical status of the target (lexical decision), followed by judging whether the prime and target were produced by the same speaker or different speakers (speaker discrimination). The results showed that both lexical decision and speaker discrimination were facilitated to a smaller extent when the prime and target were produced by different speakers, indicating reduced repetition priming by speaker variability. In contrast, semantic/associative priming was not affected by speaker variability. The ISI between the prime and target did not affect either type of priming. In conclusion, speaker variability affects accessing a word’s form but not its meaning, suggesting that speaker-specific information is processed at a relatively shallow level.