Multistep Criticality Search and Power Shaping in Nuclear Microreactors with Deep Reinforcement Learning

Radaideh, Majdi I.; Tunkle, Leo; Price, Dean; Abdulraheem, Kamal; Lin, Linyu; Elias, Moutaz

doi:10.1080/00295639.2024.2447012

Title: Multistep Criticality Search and Power Shaping in Nuclear Microreactors with Deep Reinforcement Learning

Journal Article · Wed Feb 12 00:00:00 UTC 2025 · Nuclear Science and Engineering

DOI: https://doi.org/10.1080/00295639.2024.2447012 · OSTI ID:2515660

^[1];

^[1]; Abdulraheem, Kamal ^[1];

^[2]; Elias, Moutaz ^[3]

Univ. of Michigan, Ann Arbor, MI (United States)
Idaho National Laboratory (INL), Idaho Falls, ID (United States)
Lam Research Corporation, Tualatin, OR (United States)

Reducing operation and maintenance costs is a key objective for advanced reactors in general and microreactors in particular. To achieve this reduction, developing robust autonomous control algorithms is essential to ensure safe and autonomous reactor operation. Recently, artificial intelligence and machine learning algorithms, specifically reinforcement learning (RL) algorithms, have seen rapid increased application to control problems, such as plasma control in fusion tokamaks and building energy management. In this work, we introduce the use of RL for intelligent control in nuclear microreactors. The RL agent is trained using Proximal Policy Optimization (PPO) and Advantage Actor-Critic (A2C), cutting-edge deep RL techniques, based on a high-fidelity simulation of a microreactor design inspired by the Westinghouse eVinci^TM design. We utilized a Serpent model to generate data on drum positions, core criticality, and core power distribution for training a feedforward neural network surrogate model. This surrogate model was then used to guide a PPO and A2C control policies in determining the optimal drum position across various reactor burnup states, ensuring critical core conditions and symmetrical power distribution across all six core portions. The results demonstrate the excellent performance of PPO in identifying optimal drum positions, achieving a hexant power tilt ratio of approximately 1.002 (within the limit of <1.02), and maintaining criticality within a 10 pcm range. A2C did not provide as competitive of a performance as PPO in terms of performance metrics for all burnup steps considered in the cycle. Additionally, the results highlight the capability of well-trained RL control policies to quickly identify control actions, suggesting a promising approach for enabling real-time autonomous control through digital twins.

View Journal Article

Cite

Export

Save

Research Organization:: Idaho National Laboratory (INL), Idaho Falls, ID (United States)

Sponsoring Organization:: USDOE; USDOE Laboratory Directed Research and Development (LDRD) Program; USDOE Office of Nuclear Energy (NE)

Grant/Contract Number:: AC07-05ID14517; NE0009424

OSTI ID:: 2515660

Report Number(s):: INL/JOU--24-78765-Rev000

Journal Information:: Nuclear Science and Engineering, Journal Name: Nuclear Science and Engineering Journal Issue: sup1 Vol. 200; ISSN 0029-5639

Publisher:: Taylor & FrancisCopyright Statement

Country of Publication:: United States

Language:: English

References (22)

TD-Gammon: A Self-Teaching Backgammon Program Tesauro, Gerald Applications of Neural Networks https://doi.org/10.1007/978-1-4757-2379-3_11	book	January 1995
Simple statistical gradient-following algorithms for connectionist reinforcement learning Williams, Ronald J. Machine Learning, Vol. 8, Issue 3-4 https://doi.org/10.1007/BF00992696	journal	May 1992
The Serpent Monte Carlo code: Status, development and applications in 2013 Leppänen, Jaakko; Pusa, Maria; Viitanen, Tuomas Annals of Nuclear Energy, Vol. 82 https://doi.org/10.1016/j.anucene.2014.08.024	journal	August 2015
Large-scale design optimisation of boiling water reactor bundles with neuroevolution Radaideh, Majdi I.; Forget, Benoit; Shirvan, Koroush Annals of Nuclear Energy, Vol. 160 https://doi.org/10.1016/j.anucene.2021.108355	journal	September 2021
Multilayer perception based reinforcement learning supervisory control of energy systems with application to a nuclear steam supply system Dong, Zhe; Huang, Xiaojin; Dong, Yujie Applied Energy, Vol. 259 https://doi.org/10.1016/j.apenergy.2019.114193	journal	February 2020
Reinforced model predictive control (RL-MPC) for building energy management Arroyo, Javier; Manna, Carlo; Spiessens, Fred Applied Energy, Vol. 309 https://doi.org/10.1016/j.apenergy.2021.118346	journal	March 2022
Neural-based time series forecasting of loss of coolant accidents in nuclear power plants Radaideh, Majdi I.; Pigg, Connor; Kozlowski, Tomasz Expert Systems with Applications, Vol. 160 https://doi.org/10.1016/j.eswa.2020.113699	journal	December 2020
Deep reinforcement learning for a multi-objective operation in a nuclear power plant Bae, Junyong; Kim, Jae Min; Lee, Seung Jun Nuclear Engineering and Technology, Vol. 55, Issue 9 https://doi.org/10.1016/j.net.2023.06.009	journal	September 2023
Physics-informed reinforcement learning optimization of nuclear assembly design Radaideh, Majdi I.; Wolverton, Isaac; Joseph, Joshua Nuclear Engineering and Design, Vol. 372 https://doi.org/10.1016/j.nucengdes.2020.110966	journal	February 2021
NEORL: NeuroEvolution Optimization with Reinforcement Learning—Applications to carbon-free energy systems Radaideh, Majdi I.; Du, Katelin; Seurin, Paul Nuclear Engineering and Design, Vol. 412 https://doi.org/10.1016/j.nucengdes.2023.112423	journal	October 2023
UO2-fueled microreactors: Near-term solutions to emerging markets Shirvan, Kororush; Buongiorno, Jacopo; MacDonald, Ruaridh Nuclear Engineering and Design, Vol. 412 https://doi.org/10.1016/j.nucengdes.2023.112470	journal	October 2023
Thermal Modeling of an eVinci™-like heat pipe microreactor using OpenFOAM Price, Dean; Roskoff, Nathan; Radaideh, Majdi I. Nuclear Engineering and Design, Vol. 415 https://doi.org/10.1016/j.nucengdes.2023.112709	journal	December 2023
Control automation in the heat-up mode of a nuclear power plant using reinforcement learning Park, JaeKwan; Kim, TaekKyu; Seong, SeungHwan Progress in Nuclear Energy, Vol. 145 https://doi.org/10.1016/j.pnucene.2021.104107	journal	March 2022
Surrogate modeling of advanced computer simulations using deep Gaussian processes Radaideh, Majdi I.; Kozlowski, Tomasz Reliability Engineering & System Safety, Vol. 195 https://doi.org/10.1016/j.ress.2019.106731	journal	March 2020
Human-level control through deep reinforcement learning Mnih, Volodymyr; Kavukcuoglu, Koray; Silver, David Nature, Vol. 518, Issue 7540 https://doi.org/10.1038/nature14236	journal	February 2015
Magnetic control of tokamak plasmas through deep reinforcement learning Degrave, Jonas; Felici, Federico; Buchli, Jonas Nature, Vol. 602, Issue 7897 https://doi.org/10.1038/s41586-021-04301-9	journal	February 2022
Deep Reinforcement Learning Control of a Boiling Water Reactor Chen, Xiangyi; Ray, Asok IEEE Transactions on Nuclear Science, Vol. 69, Issue 8 https://doi.org/10.1109/TNS.2022.3187662	journal	August 2022
Demonstration of Microreactor Reactivity Control with Reinforcement Learning Radaideh, Majdi; Price, Dean; Abdulraheem, Kamal International Conference on Physics of Reactors (PHYSOR 2024) https://doi.org/10.13182/PHYSOR24-43480	conference	January 2024
Playing Atari with Deep Reinforcement Learning Mnih, Volodymyr; Kavukcuoglu, Koray; Silver, David arXiv https://doi.org/10.48550/arXiv.1312.5602	preprint	January 2013
High-Dimensional Continuous Control Using Generalized Advantage Estimation Schulman, John; Moritz, Philipp; Levine, Sergey arXiv https://doi.org/10.48550/arXiv.1506.02438	preprint	January 2015
OpenAI Gym Brockman, Greg; Cheung, Vicki; Pettersson, Ludwig arXiv https://doi.org/10.48550/arXiv.1606.01540	preprint	January 2016
Proximal Policy Optimization Algorithms Schulman, John; Wolski, Filip; Dhariwal, Prafulla arXiv https://doi.org/10.48550/arXiv.1707.06347	preprint	January 2017

Similar Records

Nuclear microreactor transient and load-following control with deep reinforcement learning

Journal Article · Thu Jun 19 00:00:00 UTC 2025 · Energy Conversion and Management. X · OSTI ID:3027754

Tunkle, Leo; Abdulraheem, Kamal; Lin, Linyu; +1 more

Enhancing Autonomous Control of Microreactors Using Multi-Agent Reinforcement Learning

Conference · Sun Jun 15 00:00:00 UTC 2025 · OSTI ID:3030240

Tunkle, Leo; Abdulraheem, Kamal; Radaideh, Majdi; +1 more

Decentralized Distributed Proximal Policy Optimization (DD-PPO) for High Performance Computing Scheduling on Multi-User Systems

Journal Article · Tue May 06 00:00:00 UTC 2025 · The Journal of Supercomputing · OSTI ID:2575548

Sgambati, Matthew R; Anderson, Matthew William; Vakanski, Aleksandar

Related Subjects

22 GENERAL STUDIES OF NUCLEAR REACTORS
Advantage Actor-Critic
criticality search
microreactors
neural networks
proximal policy optimization
reactivity control
reinforcement learning

Title: Multistep Criticality Search and Power Shaping in Nuclear Microreactors with Deep Reinforcement Learning

Citation Formats

References (22)

Similar Records

Related Subjects