Deep reinforcement learning-based strategy for maximizing returns from renewable energy and energy storage systems in multi-electricity markets

The integration of Renewable Energy Sources (RES) with Energy Storage Systems (ESS) presents challenges and opportunities in optimizing their participation in electricity markets. This study introduces a novel approach that leverages Deep Reinforcement Learning (RL) algorithms to develop optimal bidding strategies for collocated RES with Battery ESS (BESS) configurations, enabling multi-market participation in both energy and ancillary services (AS) markets. The proposed method uses a Markov Decision Process (MDP) framework to manage BESS utilization dynamically, considering market conditions and technical constraints. As an RL agent, the actor-critic approach known as the Twin Delayed Deep Deterministic (TD3) Policy Gradient algorithm is implemented. A data-driven training process facilitates model learning while minimizing the required training dataset and time. Focused on the Irish context, the case study involves participation in both the day-ahead energy market and reserve services for frequency droop curve response of the DS3 Programme. Historical data from a 7 MW solar PV plant and a 1 MWh BESS are utilized to evaluate the performance. The RL agent dynamically adapts to market dynamics and system constraints, achieving substantial economic benefits compared to benchmark strategies, with an additional 8271<euro>, 166,738<euro>, and 11,369<euro>, respectively.