Human listeners can judge the temporal order of acoustic events quite accurately in certain conditions. We hypothesized that this accuracy is realized by coding temporal order as a single neural code of spectral change in the auditory system. To test this hypothesis, we examined whether adaptation to linear frequency glides affects subsequent temporal order judgment of brief tones. We found that the point of subjective simultaneity between two tones shifted depending on the direction of spectral change of the adaptor, as predicted by the hypothesis. The amount of aftereffect was significantly reduced when adaptor and test tones were presented in different frequency regions or to different ears, suggesting that the relevant neural units exist predominantly in a frequency-selective monaural pathway.