Hierarchical multi-agent drl-based secondary control for real-time voltage and frequency regulation in renewable-integrated microgrids

Abstract

The integration of inverter-based microgrids (MGs) often induces variations in system characteristics and disturbances in the network. In islanded operation, voltage and frequency oscillations arise due to the lack of synchronization with the main grid. Ensuring reliable operation under such conditions requires advanced control techniques and proactive monitoring. This paper proposes an intelligent multi-agent secondary control architecture based on the Deep Deterministic Policy Gradient (DDPG) algorithm, where reinforcement learning (RL) agents dynamically adjust the parameters of proportional–integral (PI) controllers to mitigate disturbances and adapt to system uncertainties and varying operating conditions. The proposed framework employs a two-layer hierarchical structure. At the lower layer, two independent RL agents regulate frequency and voltage by tuning PI controller gains in real-time, ensuring stable operation under fluctuating loads and distributed generation. The upper layer incorporates a supervisory agent that monitors harmonic distortion and computes corrective signals sent to the droop controller, enabling coordinated adaptation and enhancing dynamic stability by reducing transient oscillations. As a proof-of-concept, the proposed multi-agent framework is implemented on a single DG unit within a four-converter microgrid, demonstrating the potential for improved dynamic performance under realistic operational scenarios. Validation in MATLAB/Simulink demonstrates substantial improvements in system performance. Frequency settling time decreased from 0.19 s (at Kp=0.3) to 0.10 s (at Kp=0.1), and voltage settling time decreased from 0.20 s (at Kp=0.3) to 0.13 s (at Kp=0.1). Total harmonic distortion (THD) significantly reduced from 2.07% (at Kp=0.3) and 2.03% (at Kp=0.2) to 0.68% (at Kp=0.1). These results highlight the scalability and effectiveness of the proposed multi-agent DDPG-based secondary control, offering a reliable and intelligent solution for advanced MG energy management under realistic operating conditions.