SOAC: Supervised Off-Policy Actor -Critic for Recommender Systems

Improving users' long-term experience in recommender systems (RS) has become a growing concern for recommendation platforms. Reinforcement learning (RL) is an attractive approach because it can plan and optimize long-term returns sequentially. However, directly applying RL as an online learning method in the RS setting can significantly compromise users' satisfaction and experience. As a result, learning the recommendation policy from logged feedback collected under different policies has emerged as a promising direction. Offline learning enables the agent to utilize off-policy learning techniques. Nevertheless, several challenges need to be addressed, such as distribution shift. In this paper, we propose a novel RL method, called Supervised Off-Policy Actor-Critic (SOAC), for learning the recommendation policy from the logged feedback without exploration. The proposed SOAC addresses challenges, including distribution shift and extrapolation errors, and focuses on improving the ranking of items in a recommendation list. The experimental results demonstrate that SOAC can achieve better recommendation performance than existing supervised RL methods.