Safe Building HVAC Control via Batch Reinforcement Learning

In this paper, we study safe building HVAC control via batch reinforcement learning. Random exploration in building HVAC control is infeasible due to safety considerations. However, diverse states are necessary for RL algorithms to learn useful policies. To enable safety during exploration, we propose guided exploration by adding a Gaussian noise to a hand-crafted rule-based controller. Adjusting the variance of the noise provides a tradeoff between the diversity of the dataset and the safety. We apply Conservative Q Learning (CQL) to learn a policy. CQL ensures that the trained policy stays within the policy distribution used to collect the dataset, thereby guarantees safety at deployment. To select the optimal policy during the offline training, we apply model-based performance evaluation. We use the widely adopted CityLearn testbed to evaluate the performance of our proposed method. Compared with a rule-based controller, our approach obtains 12% similar to 35% reduction in ramping, 3% similar to 10% reduction in 1-load factor, 3% similar to 8% reduction in daily peak at deployment with less than 10% performance degradation during the exploration. On the contrary, the performance degradation of the state-of-the-art online reinforcement learning algorithm during exploration is around 8% 18%. It also fails to surpass the performance of the rule-based controller at deployment.