Localization of a sound source is in essence the act of decoding the directional information with which the sound was endowed by the head and ears upon measurement by the cochlea. Yet, as the source's directional signature is conflated with the spectral characteristics of the source and the latter is often not known to the listener, this directional signature may be obscured, hampering localization. Current localization models generally avoid this problem by considering sources whose spectrum is known to the listener. In this paper, we investigate how an ideal-observer would deal with this uncertainty of the source: by means of a prior on the source spectrum built from previous experiences. To this end, an ecologically valid prior was constructed from databases of environmental sounds and speech. Incorporation of this prior allowed to explain the results of a localization experiment in which the stimulus was varied, without any parameter fitting. It was shown that if the spectrum of the source deviates too much from those of real-world environments, this results in localization errors, because the source does not fit the prior used by the listener. Moreover, it seems that the binaural spectral gradient contains the relevant spectral information and that the ipsilateral side has more weight in the decision. We could not corroborate the experimental indication that only the positive spectral gradient values are used for localization. Finally, the model including the ecologically valid prior was also better in explaining the experimental data on localization of invariably flat spectrum stimuli, allowing for the possibility that human listeners may rather use a multi-purpose than a situation-specific spectral prior.