Q-Learning based Edge Caching Optimization for D2D Enabled Hierarchical Wireless Networks

Caching at the edge of mobile networks can significantly offload network traffic while satisfying content requests from mobile users locally. The contents can be requested from the proximity users via Device-to-device (D2D) communications while proactive caching the popular content to local users. However, the assumptions that content popularity is equal to user preference in several existing studies, which are invalid and not rigorous due to the fact that content popularity is calculated by the statistic of user requests within a certain period while user preference reflects the probability of a content requested by the individual user. Motivated by this, in this paper, we study the edge caching optimization of hierarchical wireless networks. Our aiming is to maximize the size of content offload by D2D communications. In particular, the edge caching policy with D2D sharing model based on the analysis of user mobility and social relationship is derived. We first prove the problem is NP-hard and then formulate it as a Markov Decision Process (MDP) problem, finally a Q-learning based distributed content replacement strategy is proposed. The large-scale real trace based experiment results show the effectiveness of our proposed framework.