Accelerated Primal-Dual Deep Reinforcement Learning for Efficient Energy Management of Hybrid Electric Vehicles

This research aims to devise an effective energy management strategy (EMS) to enhance the fuel efficiency of power-split type hybrid electric vehicles. Utilizing advancements in reinforcement learning (RL), the study introduces a novel EMS strategy that fuses deep deterministic policy gradient (DDPG) and accelerated primal-dual optimization (APDO), resulting in accelerated primal-dual deep deterministic policy gradient (APD3). Addressing issues like overestimated values and slow convergence in traditional deep RL (DRL) methods, a cutting-edge APD3 algorithm enhances the learning performance of EMS. APD3 employs a dual-critic structure to simultaneously update primal and dual variables. Moreover, it integrates an off-policy trained dual variable (lambda\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\lambda$$\end{document}) update process to enhance sampling efficiency and expedite the dual variable search process, thereby aiding in optimal action selection for the DRL agent. The overall performance under the New European Driving Cycle (NEDC), encompassing fuel economy, convergence speed, and robustness, is investigated. Simulation results illustrate that APD3 control reduces total fuel consumption by 6.78% and 7.75% compared to DDPG and TD3, respectively. Additionally, the fuel economy of the EMS system based on DDPG and TD3 reaches 81.8% and 87.7%, while APD3 achieves 95.1% of DP. The adaptability of the APD3-based EMS is further assessed with combined unknown test drive cycle under realistic city and highway conditions. Moreover, the robustness is evaluated with varying initial values of state of charge, fostering the advancement of a sustainable transportation system.