CTDE Multi-Agent Deep Q-Network for Grid-Interactive Energy Management of a Saharan Microgrid

REDOUANE LEKBIR MIHRAMANE, SIDI SALAH ECH-CHARQAOUY, Abdelkader BOULEZHAR, Nizar ECH-CHARQAOUY, AMJAD ECH-CHARQAOUY

Abstract


We propose a grid-interactive, decentralized multi-agent Deep Q-Network (MA-DQN) with Centralized Training, Decentralized Execution for energy management in a Saharan hybrid microgrid (Boujdour, Morocco). Independent agents supervise PV, wind, diesel and batteries; a low-overhead PCC coordinator enforces import/export exclusivity, zero-crossing deadband and dwell time. A multi-objective reward balances cost, CO? and grid exchange, with penalties on ±10% voltage and SOC violations. On a 24-h case (demand 991.1 kWh), the learned policy delivers renewables-first dispatch (PV 56.4%, diesel 34.4%, grid 9.2%), battery throughput ±7.6 kWh, and 272.5 kg CO? (~275 g/kWh). Versus a PSO baseline, cost falls 1250?1120 MAD (?10.4%), CO? 318.9?272.5 kg (?14.6%), and inference latency 12.8?2.1 s (?83.6%). Sensitivity on the CO? weight ? yields tunable cost–carbon trade-offs (? 0.1?2.0: 310?240 kg CO? at +15.5% cost). Stress tests (PV depression/outage, +20% evening peak) maintain supply and power-quality; ablation confirms reduced PCC chattering versus single-agent DQN.


Keywords


Microgrid energy management; reinforcement learning; multi-agent DQN; CTDE; demand response.

Full Text:

PDF

References


Q. Liu, K. Kamoto, and X. Liu, “Microgrids-as-a-service for rural electrification in Sub-Saharan Africa,” Computers, Materials & Continua, vol. 63, pp. 1249–1261, 2020, doi: 10.32604/cmc.2020.05598.

A. Giedraityte, S. Rimkevicius, M. Marciukaitis, V. Radziukynas, and R. Bakas, “Hybrid renewable energy systems: a review of optimization approaches and future challenges,” Applied Sciences, vol. 15, no. 4, pp. 1744, 2025, doi: 10.3390/app15041744.

J. Hirwa, A. Zolan, W. Becker, T. Flamand, and A. Newman, “Optimizing design and dispatch of a resilient renewable energy microgrid for a South African hospital,” Applied Energy, vol. 348, pp. 121438, 2023, doi: 10.1016/j.apenergy.2023.121438.

Q. Hassan, S. Algburi, A. Z. Sameen, H. M. Salman, and M. Jaszczur, “A review of hybrid renewable energy systems: solar and wind-powered solutions: challenges, opportunities, and policy implications,” Results in Engineering, vol. 20, pp. 101621, 2023, doi: 10.1016/j.rineng.2023.101621.

H. Al-Rawashdeh, O. A. Al-Khashman, J. T. Al Bdour, M. R. Gomaa, H. Rezk, A. Marashli, L. M. Arrfou, and M. Louzazni, “Performance analysis of a hybrid renewable-energy system for green buildings to improve efficiency and reduce GHG emissions with multiple scenarios,” Sustainability, vol. 15, no. 9, pp. 7529, 2023, doi: 10.3390/su15097529.

M. Ali, K. Prakash, M. A. Hossain, and H. R. Pota, “Intelligent energy management: evolving developments, current challenges, and research directions for sustainable future,” Journal of Cleaner Production, vol. 314, pp. 127904, 2021, doi: 10.1016/j.jclepro.2021.127904.

International Energy Agency, “Global Energy Review 2025: CO? Emissions,” IEA, Paris, France, 2025. https://www.iea.org/reports/global-energy-review-2025/co2-emissions.

R. Thapa-Parajuli, R. Neupane, M. Timsina, B. Pokharel, D. Poudel, M. Maharjan, S. P. KC, and S. Shrestha, “Global CO? emission reduction disparities after and before COVID-19,” Sustainability, vol. 17, no. 14, pp. 6602, 2025, doi: 10.3390/su17146602.

I. T. Halidou, H. O. R. Howlader, M. M. Gamil, M. H. Elkholy, and T. Senjyu, “Optimal power scheduling and techno-economic analysis of a residential microgrid for a remotely located area: A case study for the Sahara Desert of Niger,” Energies, vol. 16, no. 8, pp. 3471, 2023, doi: 10.3390/en16083471.

L. Richard, C. Boudinet, S. A. Ranaivoson, J. O. Rabarivao, A. E. Befeno, D. Frey, M.-C. Alvarez-Hérault, B. Raison, and N. Saincy, “Development of a DC microgrid with decentralized production and storage: From the lab to field deployment in Rural Africa,” Energies, vol. 15, no. 18, pp. 6727, 2022, doi: 10.3390/en15186727.

R. L. Mihramane, S. S. Ech-Charqaouy, D. Saifaoui, N. Ech-Charqaouy, and A. Ech-Charqaouy, “Management and optimization of a renewable energy hybrid system integrated into a microgrid, considering constraints related to voltage stability and service continuity,” IJRER, vol. 15, no. 2, pp. 213–225, 2025, doi: 10.20508/ijrer.v15i2.14781.g9042.

S. S. Ech-Charqaouy, D. Saifaoui, O. Benzohra, and A. Lebsir, “Integration of decentralized generations into the distribution network smart grid downstream of the meter,” IJsmartGrid, vol. 4, no. 1, pp. 17–27, 2020, doi: 10.20508/ijsmartgrid.v4i1.92.g78.

S. S. Ech-Charqaouy, D. Saifaoui, O. Benzohra, and A. Lebsir, “Impact of integrating renewable energies into distribution networks on the voltage profile,” IJRER, vol. 10, no. 1, pp. 143–154, 2020, doi: 10.20508/ijrer.v10i1.10204.g7848.

E. Sutanto, P. E. Astawa, F. Fahmi, M. I. Hamid, M. Yazid, W. Shalannanda, and M. Aziz, “Lithium-ion battery state-of-charge estimation from the voltage discharge profile using gradient vector and support vector machine,” Energies, vol. 16, no. 3, Art. no. 1083, 2023, doi: 10.3390/en16031083.

J. A. Braun, R. Behmann, D. Schmider, and W. G. Bessler, “State of charge and state of health diagnosis of batteries with voltage-controlled models,” Journal of Power Sources, vol. 544, Art no. 231828, 2022, doi: 10.1016/j.jpowsour.2022.231828.

L. Yi, P. Langlotz, M. Hussong, M. Glatt, F. Sousa, and J. C. Aurich, “An integrated energy management system using double deep Q-learning and energy storage equipment to reduce energy cost in manufacturing under real-time pricing condition: A case study of a scale-model factory,” CIRP Journal of Manufacturing Science and Technology, vol. 38, pp. 844–860, 2022, doi: 10.1016/j.cirpj.2022.07.009.

E. Foruzan, L.-K. Soh, and S. Asgarpoor, “Reinforcement learning approach for optimal distributed energy management in a microgrid,” IEEE Transactions on Power Systems, vol. 33, no. 5, pp. 5749–5758, 2018, doi: 10.1109/TPWRS.2018.2823641.

D. Liu, C. Zang, P. Zeng, W. Li, X. Wang, Y. Liu, and S. Wang, “Deep reinforcement learning for real-time economic energy management of microgrid system considering uncertainties,” Frontiers in Energy Research, vol. 11, Art no. 1163053, 2023, doi: 10.3389/fenrg.2023.1163053.

T. Levent, P. Preux, E. Le Pennec, J. Badosa, G. Henri, and Y. Bonnassieux, “Energy Management for Microgrids: A Reinforcement Learning Approach,” in Proc. IEEE PES, Bucharest, Romania, 2019, pp. 1-5, doi: 10.1109/ISGTEurope.2019.8905538.

J. Terven, “Deep reinforcement learning: A chronological overview and methods,” AI, vol. 6, no. 3, Art no. 46, 2025, doi: 10.3390/ai6030046.

U. K., K. Gurunadh, N. K. S. Keerthan, N. K. Musunuru, and A. J., “Energy management system-deep reinforcement learning based dueling DQN (Deep Q-Networks),” in Proc. Asia Pacific Conference on Innovation in Technology (APCIT), Mysore, India, 2024, pp. 1–7, doi: 10.1109/APCIT62007.2024.10673561.

E. O. Arwa and K. Folly, “Reinforcement learning techniques for optimal power control in grid-connected microgrids: A comprehensive review,” IEEE Access, vol. 8, pp. 227747–227766, 2020, doi: 10.1109/ACCESS.2020.3038735.

M. Ahmed, C. P. Lim, and S. Nahavandi, “A deep Q-network reinforcement learning-based model for autonomous driving,” in Proc. IEEE International Conference on Systems, Man, and Cybernetics (SMC), Melbourne, Australia, 2021, pp. 739-744, doi: 10.1109/SMC52423.2021.9658892.

G. Custodio, L. F. Ochoa, T. Alpcan, F. C. L. Trindade, and R. A. de Godoy Rosolen, “Using dueling double Q-learning for voltage regulation in PV-rich distribution networks,” in Proc. IEEE Madrid PowerTech, Madrid, Spain, 2021, pp. 1-6, doi: 10.1109/PowerTech46648.2021.9494903.

H. Qiu and F. Liu, “A state representation dueling network for deep reinforcement learning,” in Proc. IEEE 32nd International Conference on Tools with Artificial Intelligence (ICTAI), Baltimore, MD, USA, 2020, pp. 669–674, doi: 10.1109/ICTAI50040.2020.00107.

P. Kiran and K. R. M. Vijaya Chandrakala, “New interactive agent based reinforcement learning approach towards smart generator bidding in electricity market with microgrid integration,” Applied Soft Computing, vol. 97, Art no. 106762, 2020, doi: 10.1016/j.asoc.2020.106762.

S. Mischos, E. Dalagdi, and D. Vrakas, “Intelligent energy management systems: A review,” Artificial Intelligence Review, vol. 56, pp. 1–40, 2023, doi: 10.1007/s10462-023-10441-3.

D. M. S. Zekrifa, R. D. Prakash, M. Dhanalakshmi, R. Puviarasi, C. S. Sundar Ganesh, J. Samson Isaac, P. Duraipandy, and A. Rajaram, “Optimized controller design for renewable energy systems by using deep reinforcement learning technique,” IJRER, vol. 14, no. 1, pp. 101-110, 2024, doi:10.20508/ijrer.v14i1.14273.g8866.

P. I. N. Barbalho, A. L. Moraes, V. A. Lacerda, P. H. A. Barra, R. A. S. Fernandes, and D. V. Coury, “Reinforcement Learning Solutions for Microgrid Control and Management: A Survey,” IEEE Access, vol. 13, pp. 1–31, 2025, doi: 10.1109/ACCESS.2025.3546578.

C. Sauer, F. J. D. Lange, M. Thurow, I. Dormuth, and A.-L. Boulesteix, “Statistical parametric simulation studies based on real data,” Statistics Methodology, 2025, doi: 10.48550/arXiv.2504.04864.

N. Singh, A. Agarwal, and J. Kumar, “A decentralised power management strategy in a hybrid energy storage supported islanded residential sustainable DC microgrid system,” IJETP, vol. 18, no. 1, pp. 1–22, 2022, doi: 10.1504/IJETP.2022.10045782.

A. Iosifidis and A. Tefas, Deep learning for robot perception and cognition. cambridge, MA, USA: Academic Press, 2022.

H. Alavizadeh, H. Alavizadeh, and J. Jang-Jaccard, “Deep Q-learning based reinforcement learning approach for network intrusion detection,” Computers, vol. 11, no. 3, Art no. 41, 2022, doi: 10.3390/computers11030041.

J. Kumar, A. Agarwal, and N. Singh, “A Cost-Effective DC microgrid system for resilient power supply to grid-connected societies,” International Journal of Numerical Modelling: Electronic Networks, Devices and Fields, vol. 36, no. 5, pp. 1–24, 2023, doi: 10.1002/jnm.3097.

T. Nakabi and P. Toivanen, “Deep reinforcement learning for energy management in a microgrid with flexible demand,” Sustainable Energy, Grids and Networks, vol. 25, Art no. 100413, 2021, doi: 10.1016/j.segan.2020.100413.

S. Stavrev and D. Ginchev, “Reinforcement learning techniques in optimizing energy systems,” Electronics, vol. 13, no. 8, Art no. 1459, 2024, doi: 10.3390/electronics13081459.

H. Song, Y. Liu, J. Zhao, J. Liu, and G. Wu, “Prioritized replay dueling DDQN based grid-edge control of community energy storage system,” IEEE Transactions on Smart Grid, vol. 12, no. 6, pp. 4950–4961, 2021, doi: 10.1109/TSG.2021.3099133.

E. Samadi, A. Badri, and R. Ebrahimpour, “Q-learning approach for optimal power dispatch of microgrid,” in Proc. 28th Iranian Conference on Electrical Engineering (ICEE), Tabriz, Iran, 2020, pp. 1-5, doi: 10.1109/ICEE50131.2020.9260722.

A. Akta? and Y. Kirçiçek, Solar Hybrid Systems. Cambridge, MA, USA: Academic Press, 2021.

A. Kumari and S. Tanwar, “Reinforcement learning for multiagent-based residential energy management system,” in Proc. IEEE Globecom Workshops (GC Wkshps), Madrid, Spain, 2021, pp. 1–6, doi: 10.1109/GCWkshps52748.2021.9682182.

M. K. Perera, K. T. M. U. Hemapala, and W. D. A. S. Wijayapala, “Developing a reinforcement learning model for energy management of microgrids in python,” in Proc. International Conference on Computational Intelligence and Knowledge Economy (ICCIKE), Dubai, United Arab Emirates, 2021, pp. 68–73, doi: 10.1109/ICCIKE51210.2021.9410754.




DOI (PDF): https://doi.org/10.20508/ijsmartgrid.v10i2.562.g426

Refbacks

  • There are currently no refbacks.


www.ijsmartgrid.com; www.ijsmartgrid.org

ilhcol@gmail.com; icolak@gazi.edu.tr

Online ISSN: 2602-439X

Publisher: ilhami COLAK

https://www.ilhamicolak.org/english.htm

Cited in SCOPUS Q1, SCIMAGO, Google Scholar and CrossRef

 

 

LinkedIn Logo Follow us on LinkedIn

Google Scholar Statistics of ijSmartGrid

https://scholar.google.com/citations?user=qP9ZUKAAAAAJ&hl=tr&authuser=1 

 

SCIMAGO information:

https://www.scimagojr.com/journalsearch.php?q=21101271431&tip=sid&clean=0#google_vignette

Q2, h-index:13