Breast cancer is a deadly disease which is most commonly diagnosed in women. It spreads worldwide as the number of cases is increasing each day. The significant increase in cancer cases leads researchers to develop imaging tools for its detection. However, false-positive rates in manual detections are more, which may be due to human error, time taking process or some other issues. If these cancers are not detected at an early stage, they can cause the death of the patient. Therefore, several machine learning and deep learning-based methodologies have come into existence, which can be used to detect breast cancer in early stages. Nowadays, deep learning-based methods are trending, therefore, this paper discusses some important work done by researchers using deep learning. This survey provides an in-depth study of various deep learning models used for breast cancer detection. The details about mammography, histopathology and ultrasound datasets are also given which are being used for comparative analysis of the developed methods. This survey is not only on one imaging modality instead covering three modalities and thus giving a wider study of various methods. Important works from various years have been discussed along with recent works based on vision transformer. Therefore, this paper presents all the necessary information to researchers working in this area and they can think of new ideas based on it to carry out further work.