Ambient vibration tests have been gaining more and more popularity in structural condition monitoring because they are easy to carry out and cause only minimum disturbance to the normal operation of the tested structures. Compared to the forced vibration test, the input force is not known in ambient vibration test. The ambient excitation sources may come from wind, sea wave, traffic, and other natural sources and human activities. Common approaches to extract structural vibration properties from ambient vibration tests are based on the assumption that the ambient excitation force is a white noise. In reality, the unknown excitation forces may not be white noise, or even not a wide band process, and inevitably contain noises. In this paper, the reliability of using ambient vibration tests to extract structural vibration properties is investigated. Vibration test data from the ASCE/IASC benchmark model obtained from three excitation sources, namely impact hammer, shaker with random white noise excitations and ambient excitation (traffic and human activities) are analyzed. The extracted vibration properties (frequency, mode shape and damping) are compared. A numerical model is also created to simulate dynamic testing. Numerically simulated data is smeared with different types of noises to investigate the influence of noises on extracted structural vibration parameters. Discussions are made on the accuracy of using ambient vibration tests to extract structural vibration properties. The conditions under which the ambient vibration tests lead to accurate structural property extraction are also defined.