Like any other biometric systems, Automatic Speaker Verification (ASV) systems are also vulnerable to the spoofing attacks. Hence, it is important to develop the countermeasures in order to handle these attacks. In spoofing mainly two types of attacks are considered, logical access attacks and presentation attacks. In the last few decades, several systems have been proposed by various researchers for handling these kinds of attacks. However, noise handling capability of ASV systems is of major concern, as the presence of noise may make an ASV system to falsely evaluate the original human voice as the spoofed audio. Hence, the main objective of this paper is to review and analyze the various noise robust ASV systems proposed by different researchers in recent years. The paper discusses the various front end and back-end approaches that have been used to develop these systems with putting emphasis on the noise handling techniques. Various kinds of noises such as babble, white, background noises, pop noise, channel noises etc. affect the development of an ASV system. This survey starts with discussion about the various components of ASV system. Then, the paper classifies and discusses various enhanced front end feature extraction techniques like phase based, deep learning based, magnitude-based feature extraction techniques etc., which have been proven to be robust in handling noise. Secondly, the survey highlights the various deep learning and other baseline models that are used in backend, for classification of the audio correctly. Finally, it highlights the challenges and issues that still exist in noise handling and detection, while developing noise robust ASV systems. Therefore, on the basis of the proposed survey it can be interpreted that the noise robustness of ASV system is the challenging issue. Hence the researchers should consider the robustness of ASV against noise along with spoofing attacks.