TTL-Based Cache Utility Maximization Using Deep Reinforcement Learning

Utility-driven caching opened up a new design opportunity for caching algorithms by modeling the admission and eviction control as a utility maximization process with essential support for service differentiation. Nevertheless, there is still to go in terms of adaptability to changing environment. Slow convergence to an optimal state may degrade actual user-experienced utility, which gets even worse in non-stationary scenarios where cache control should be adaptive to time-varying content request traffic. This paper proposes to exploit deep reinforcement learning (DRL) to enhance the adaptability of utility-driven time-to-live (TTL)-based caching. Employing DRL with long short-term memory helps a caching agent learn how it adapts to the temporal correlation of content popularities to shorten the transient-state before the optimal steady-state. In addition, we elaborately design the state and action spaces of DRL to overcome the curse of dimensionality, which is one of the most frequently raised issues in machine learning-based approaches. Experimental results show that policies trained by DRL can outperform the conventional utility-driven caching algorithm under some non-stationary environments where content request traffic changes rapidly.