Deep reinforcement learning based optimization for a tightly coupled nuclear renewable integrated energy system

Yi, Zonggen; Luo, Yusheng; Westover, Tyler; Katikaneni, Sravya; Ponkiya, Binaka; Sah, Suba; Mahmud, Sadab; Raker, David; Javaid, Ahmad; Heben, Michael J.; Khanna, Raghav

doi:10.1016/j.apenergy.2022.120113

Deep reinforcement learning based optimization for a tightly coupled nuclear renewable integrated energy system

Journal Article · Tue Nov 01 04:00:00 EDT 2022 · Applied Energy

DOI:https://doi.org/10.1016/j.apenergy.2022.120113· OSTI ID:1903577

^[1]; ^[1]; ^[1]; Katikaneni, Sravya ^[2]; Ponkiya, Binaka ^[2]; Sah, Suba ^[2]; ^[2]; Raker, David ^[2]; ^[2]; Heben, Michael J. ^[2]; Khanna, Raghav ^[2]

Idaho National Lab. (INL), Idaho Falls, ID (United States)
Univ. of Toledo, OH (United States)

New ways to integrate energy systems to maximize efficiency are being sought to meet carbon emissions goals. Nuclear-renewable integrated energy system (NR-IES) concepts are a leading solution that couples a nuclear power plant with renewable energy, hydrogen generation plants, and energy storage systems, such that thermal and electrical power are dispatchable to fulfill grid-flexibility requirements while also producing hydrogen and maximizing revenue. Here, this paper introduces a deep reinforcement learning (DRL)-based framework to address the complex decision-making tasks for NR-IES. The objective is to maximize revenue by generating and selling hydrogen and electricity simultaneously according to their time-varying prices while keeping the energy flow in the subsystems in balance. A Python-based simulator for a NR-IES concept has been developed to integrate with OpenAI Gym and Ray/RLlib to enable an efficient and flexible computational framework for DRL research and development. Three state-of-the-art DRL algorithms have been investigated, including two-delayed deep deterministic policy gradient (TD3), soft-actor critic (SAC), proximal policy optimization (PPO), to illustrate DRL’s superiority for controlling NR-IES by comparing it with a conventional control approach, particle swarm optimization (PSO). In this effort, PPO has shown more-stable performance and also better generalization capability than SAC and TD3. Comparisons with PSO have demonstrated that, on average, PPO can achieve 13.9% more mean episode returns from the training process and 29.4% more mean episode returns from the testing process when different hydrogen-production targets are applied.

View Accepted Manuscript (DOE)

Research Organization:: Idaho National Laboratory (INL), Idaho Falls, ID (United States)

Sponsoring Organization:: USDOE Office of Nuclear Energy (NE); USDOE Laboratory Directed Research and Development (LDRD) Program

Grant/Contract Number:: AC07-05ID14517

OSTI ID:: 1903577

Alternate ID(s):: OSTI ID: 1961678

Report Number(s):: INL/JOU-21-64881-Rev000

Journal Information:: Applied Energy, Journal Name: Applied Energy Vol. 328; ISSN 0306-2619

Publisher:: ElsevierCopyright Statement

Country of Publication:: United States

Language:: English

References (42)

A survey and critique of multiagent deep reinforcement learning Hernandez-Leal, Pablo; Kartal, Bilal; Taylor, Matthew E. Autonomous Agents and Multi-Agent Systems, Vol. 33, Issue 6 https://doi.org/10.1007/s10458-019-09421-1	journal	October 2019
Intelligent Decision-Making for Smart Home Energy Management Berlink, Heider; Kagan, Nelson; Reali Costa, Anna Helena Journal of Intelligent & Robotic Systems, Vol. 80, Issue S1 https://doi.org/10.1007/s10846-014-0169-8	journal	December 2014
Stochastic optimization of cost-risk for integrated energy system considering wind and solar power correlated Zheng, Jiehui; Kou, Yanni; Li, Mengshi Journal of Modern Power Systems and Clean Energy, Vol. 7, Issue 6 https://doi.org/10.1007/s40565-019-0519-4	journal	July 2019
Energy analysis of thermal energy storages with grid configurations Rezaie, Behnaz; Reddy, Bale V.; Rosen, Marc A. Applied Energy, Vol. 117 https://doi.org/10.1016/j.apenergy.2013.11.042	journal	March 2014
Modelling and optimization of CHP based district heating system with renewable energy production and energy storage Wang, Haichao; Yin, Wusong; Abdollahi, Elnaz Applied Energy, Vol. 159 https://doi.org/10.1016/j.apenergy.2015.09.020	journal	December 2015
An MILP formulation for the optimal management of microgrids with task interruptions Silvente, Javier; Papageorgiou, Lazaros G. Applied Energy, Vol. 206 https://doi.org/10.1016/j.apenergy.2017.08.147	journal	November 2017
Dynamic performance analysis of a high-temperature steam electrolysis plant integrated within nuclear-renewable hybrid energy systems Kim, Jong Suk; Boardman, Richard D.; Bragg-Sitton, Shannon M. Applied Energy, Vol. 228 https://doi.org/10.1016/j.apenergy.2018.07.060	journal	October 2018
Economic analysis of a nuclear hybrid energy system in a stochastic environment including wind turbines in an electricity grid Epiney, A.; Rabiti, C.; Talbot, P. Applied Energy, Vol. 260 https://doi.org/10.1016/j.apenergy.2019.114227	journal	February 2020
Evaluating the impact of multi-carrier energy storage systems in optimal operation of integrated electricity, gas and district heating networks Mirzaei, Mohammad Amin; Nazari-Heris, Morteza; Zare, Kazem Applied Thermal Engineering, Vol. 176 https://doi.org/10.1016/j.applthermaleng.2020.115413	journal	July 2020
Stochastic optimization of energy hub operation with consideration of thermal energy market and demand response Vahid-Pakdel, M. J.; Nojavan, Sayyad; Mohammadi-ivatloo, B. Energy Conversion and Management, Vol. 145 https://doi.org/10.1016/j.enconman.2017.04.074	journal	August 2017
Improvement of the cogeneration plant economy by using heat accumulator Bogdan, Z.; Kopjar, D. Energy, Vol. 31, Issue 13 https://doi.org/10.1016/j.energy.2006.01.012	journal	October 2006
Integrated scheduling of energy supply and demand in microgrids under uncertainty: A robust multi-objective optimization approach Wang, Luhao; Li, Qiqiang; Ding, Ran Energy, Vol. 130 https://doi.org/10.1016/j.energy.2017.04.115	journal	July 2017
Coordinated energy management for a cluster of buildings through deep reinforcement learning Pinto, Giuseppe; Piscitelli, Marco Savino; Vázquez-Canteli, José Ramón Energy, Vol. 229 https://doi.org/10.1016/j.energy.2021.120725	journal	August 2021
Dynamic energy dispatch strategy for integrated energy system based on improved deep reinforcement learning Yang, Ting; Zhao, Liyuan; Li, Wei Energy, Vol. 235 https://doi.org/10.1016/j.energy.2021.121377	journal	November 2021
Fossil-fuel constraints on global warming Zecca, Antonio; Chiari, Luca Energy Policy, Vol. 38, Issue 1 https://doi.org/10.1016/j.enpol.2009.06.068	journal	January 2010
Uncertainty modeling in optimal operation of energy hub in presence of wind, storage and demand response Pazouki, Samaneh; Haghifam, Mahmoud-Reza; Moser, Albert International Journal of Electrical Power & Energy Systems, Vol. 61 https://doi.org/10.1016/j.ijepes.2014.03.038	journal	October 2014
Combined heat and power system intelligent economic dispatch: A deep reinforcement learning approach Zhou, Suyang; Hu, Zijian; Gu, Wei International Journal of Electrical Power & Energy Systems, Vol. 120 https://doi.org/10.1016/j.ijepes.2020.106016	journal	September 2020
Hybrid nuclear-renewable energy systems: A review Suman, Siddharth Journal of Cleaner Production, Vol. 181 https://doi.org/10.1016/j.jclepro.2018.01.262	journal	April 2018
Aging formula for lithium ion batteries with solid electrolyte interphase layer growth Tanim, Tanvir R.; Rahn, Christopher D. Journal of Power Sources, Vol. 294 https://doi.org/10.1016/j.jpowsour.2015.06.014	journal	October 2015
Technology readiness assessment of Small Modular Reactor (SMR) designs Liu, Zhitao; Fan, Jihong Progress in Nuclear Energy, Vol. 70 https://doi.org/10.1016/j.pnucene.2013.07.005	journal	January 2014
A review on configurations, control and sizing methodologies of hybrid energy systems Upadhyay, Subho; Sharma, M. P. Renewable and Sustainable Energy Reviews, Vol. 38 https://doi.org/10.1016/j.rser.2014.05.057	journal	October 2014
Multi-agent Deep Reinforcement Learning for Distributed Energy Management and Strategy Optimization of Microgrid Market Fang, Xiaohan; Zhao, Qiang; Wang, Jinkuan Sustainable Cities and Society, Vol. 74 https://doi.org/10.1016/j.scs.2021.103163	journal	November 2021
Coherency Strain and the Kinetics of Phase Separation in LiFePO ₄ Nanoparticles Cogswell, Daniel A.; Bazant, Martin Z. ACS Nano, Vol. 6, Issue 3 https://doi.org/10.1021/nn204177u	journal	February 2012
Grandmaster level in StarCraft II using multi-agent reinforcement learning Vinyals, Oriol; Babuschkin, Igor; Czarnecki, Wojciech M. Nature, Vol. 575, Issue 7782 https://doi.org/10.1038/s41586-019-1724-z	journal	October 2019
Operation Planning of Hydrogen Storage Connected to Wind Power Operating in a Power Market Korpas, M.; Holen, A. T. IEEE Transactions on Energy Conversion, Vol. 21, Issue 3 https://doi.org/10.1109/TEC.2006.878245	journal	September 2006
Optimal Economical Schedule of Hydrogen-Based Microgrids With Hybrid Storage Using Model Predictive Control Garcia-Torres, Felix; Bordons, Carlos IEEE Transactions on Industrial Electronics, Vol. 62, Issue 8 https://doi.org/10.1109/TIE.2015.2412524	journal	August 2015
Distributed Optimal Energy Management for Energy Internet Zhang, Huaguang; Li, Yushuai; Gao, David Wenzhong IEEE Transactions on Industrial Informatics, Vol. 13, Issue 6 https://doi.org/10.1109/TII.2017.2714199	journal	December 2017
Survey of Deep Reinforcement Learning for Motion Planning of Autonomous Vehicles Aradi, Szilard IEEE Transactions on Intelligent Transportation Systems, Vol. 23, Issue 2 https://doi.org/10.1109/TITS.2020.3024655	journal	February 2022
Increasing the Flexibility of Combined Heat and Power for Wind Power Integration in China: Modeling and Implications Chen, Xinyu; Kang, Chongqing; O'Malley, Mark IEEE Transactions on Power Systems, Vol. 30, Issue 4 https://doi.org/10.1109/TPWRS.2014.2356723	journal	July 2015
Optimal Operation of Residential Energy Hubs in Smart Grids Bozchalui, Mohammad Chehreghani; Hashmi, Syed Ahsan; Hassen, Hussin IEEE Transactions on Smart Grid, Vol. 3, Issue 4 https://doi.org/10.1109/TSG.2012.2212032	journal	December 2012
An Online Optimal Dispatch Schedule for CCHP Microgrids Based on Model Predictive Control Gu, Wei; Wang, Zhihe; Wu, Zhi IEEE Transactions on Smart Grid, Vol. 8, Issue 5 https://doi.org/10.1109/TSG.2016.2523504	journal	September 2017
Integrated Management of Energy Resources in Residential Buildings—A Markovian Approach Nistor, Maricica; Antunes, Carlos Henggeler IEEE Transactions on Smart Grid, Vol. 9, Issue 1 https://doi.org/10.1109/TSG.2016.2549939	journal	January 2018
Thermo-Economic Optimization of Solar CCHP Using Both Genetic and Particle Swarm Algorithms Sanaye, Sepehr; Hajabdollahi, Hassan Journal of Solar Energy Engineering, Vol. 137, Issue 1 https://doi.org/10.1115/1.4027932	journal	July 2014
Particle Swarm Optimization for Single Objective Continuous Space Problems: A Review Bonyadi, Mohammad Reza; Michalewicz, Zbigniew Evolutionary Computation, Vol. 25, Issue 1 https://doi.org/10.1162/EVCO_r_00180	journal	March 2017
Reinforcement learning in robotics: A survey Kober, Jens; Bagnell, J. Andrew; Peters, Jan The International Journal of Robotics Research, Vol. 32, Issue 11 https://doi.org/10.1177/0278364913495721	journal	August 2013
Operations Optimization of Nuclear Hybrid Energy Systems Chen, Jun; Garcia, Humberto E.; Kim, Jong Suk Nuclear Technology, Vol. 195, Issue 2 https://doi.org/10.13182/NT15-130	journal	August 2016
PySwarms: a research toolkit for Particle Swarm Optimization in Python James V. Miranda, Lester The Journal of Open Source Software, Vol. 3, Issue 21 https://doi.org/10.21105/joss.00433	journal	January 2018
Distributed Optimal Economic Dispatch Based on Multi-Agent System Framework in Combined Heat and Power Systems Li, Yu-Shuai; Zhang, Hua-Guang; Huang, Bo-Nan Applied Sciences, Vol. 6, Issue 10 https://doi.org/10.3390/app6100308	journal	October 2016
Integrated Optimal Dispatch of a Rural Micro-Energy-Grid with Multi-Energy Stream Based on Model Predictive Control Zhang, Xin; Yang, Jianhua; Wang, Weizhou Energies, Vol. 11, Issue 12 https://doi.org/10.3390/en11123439	journal	December 2018
Optimal Variable Renewable Energy Generation Schedules Considering Market Prices and System Operational Constraints Imcharoenkul, Veeraya; Chaitusaney, Surachai Energies, Vol. 14, Issue 17 https://doi.org/10.3390/en14175320	journal	August 2021
Integrated Combined Heat and Power System Dispatch Considering Electrical and Thermal Energy Storage Yuan, Rongxiang; Ye, Jun; Lei, Jiazhi Energies, Vol. 9, Issue 6 https://doi.org/10.3390/en9060474	journal	June 2016
Reinforcement Learning and Its Applications in Modern Power and Energy Systems: A Review Cao, Di; Hu, Weihao; Zhao, Junbo Journal of Modern Power Systems and Clean Energy, Vol. 8, Issue 6 https://doi.org/10.35833/MPCE.2020.000552	journal	January 2020

Similar Records

Design and optimization of a modular hydrogen-based integrated energy system to maximize revenue via nuclear-renewable sources

Journal Article · Sat Nov 16 23:00:00 EST 2024 · Energy · OSTI ID:2503932

Hybrid-RL-MPC4CLR (Hybird-Reinforcement-Learning-Model-Predictive-Control-for-Reserve-Policy-Assisted-Critical-Load-Restoration-in-Distribution-Grids)

Software · Thu Mar 17 20:00:00 EDT 2022 · OSTI ID:code-72315

Federated Deep Reinforcement Learning for Decentralized VVO of BTM DERs

Conference · Tue Oct 01 00:00:00 EDT 2024 · OSTI ID:2477510

Related Subjects

08 HYDROGEN
29 ENERGY PLANNING, POLICY, AND ECONOMY
32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATION
97 MATHEMATICS AND COMPUTING
Deep reinforcement learning
Nuclear renewable integrated energy system
Operation optimization
System control

Deep reinforcement learning based optimization for a tightly coupled nuclear renewable integrated energy system

Citation Formats

References (42)

Similar Records

Related Subjects