In recent years, deep learning (DL) models have attracted widespread concern. Due to its own characteristics, DL has been successfully applied in the fields of object detection, superresolution reconstruction, speech recognition, natural language processing, etc., bringing high efficiency to industrial production and daily life. With the Internet of Things, 6G and other new technologies have been proposed, leading to an exponential growth in data volume. DL models currently suffer from some security issues, such as privacy issues during data collection, defense issues during model training and deployment, etc. The sensitive data of users and special institutions that are directly used as training data of DL models may lead to information leakage and serious privacy problems. In addition, DL models have encountered many malicious attacks in the real world, such as poisoning attack, exploratory attack, adversarial attack, etc., which caused model security problems. Therefore, this article discusses ways of ensuring the security and data privacy of DL models under diversified attack methods and the ways of ensuring the privacy security of edge mobile devices equipped with pretrained deep neural networks. Alternatively, this article analyzes the privacy security of DL models for typical deployment platforms such as server/cloud, edge mobile device, and web browser and, then, summarizes future research direction. © 2020 IEEE.