Intelligent multi-zone residential HVAC control strategy based on deep reinforcement learning

Du, Yan; Zandi, Helia; Kotevska, Olivera; Kurte, Kuldeep; Munk, Jeffery; Amasyali, Kadir; Mckee, Evan; Li, Fangxing

doi:10.1016/j.apenergy.2020.116117

Intelligent multi-zone residential HVAC control strategy based on deep reinforcement learning

Journal Article · Wed Nov 04 23:00:00 EST 2020 · Applied Energy

DOI:https://doi.org/10.1016/j.apenergy.2020.116117· OSTI ID:1784145

Du, Yan ^[1]; ^[2]; ^[2]; Kurte, Kuldeep ^[2]; ^[2]; ^[2]; Mckee, Evan ^[1]; Li, Fangxing ^[1]

Univ. of Tennessee, Knoxville, TN (United States)
Oak Ridge National Lab. (ORNL), Oak Ridge, TN (United States)

Residential heating, ventilation, and air conditioning (HVAC) has been considered as an important demand response resource. However, the optimization of residential HVAC control is no trivial task due to the complexity of the thermal dynamic models of buildings and uncertainty associated with both occupant-driven heat loads and weather forecasts. In this paper, we apply a novel model-free deep reinforcement learning (RL) method, known as the deep deterministic policy gradient (DDPG), to generate an optimal control strategy for a multi-zone residential HVAC system with the goal of minimizing energy consumption cost while maintaining the users’ comfort. Here, the applied deep RL-based method learns through continuous interaction with a simulated building environment and without referring to any prior model knowledge. Simulation results show that compared with the state-of-art deep Q network (DQN), the DDPG-based HVAC control strategy can reduce the energy consumption cost by 15% and reduce the comfort violation by 79%; and when compared with a rule-based HVAC control strategy, the comfort violation can be reduced by 98%. In addition, experiments with different building models and retail price models demonstrate that the well-trained DDPG-based HVAC control strategy has high generalization and adaptability to unseen environments, which indicates its practicability for real-world implementation.

View Accepted Manuscript (DOE)

Research Organization:: Oak Ridge National Laboratory (ORNL), Oak Ridge, TN (United States)

Sponsoring Organization:: USDOE Office of Energy Efficiency and Renewable Energy (EERE)

Grant/Contract Number:: AC05-00OR22725

OSTI ID:: 1784145

Alternate ID(s):: OSTI ID: 23188705

Journal Information:: Applied Energy, Journal Name: Applied Energy Journal Issue: 1 Vol. 281; ISSN 0306-2619

Publisher:: ElsevierCopyright Statement

Country of Publication:: United States

Language:: English

References (29)

Building operation and energy performance: Monitoring, analysis and optimisation toolkit Costa, Andrea; Keane, Marcus M.; Torrens, J. Ignacio Applied Energy, Vol. 101 https://doi.org/10.1016/j.apenergy.2011.10.037	journal	January 2013
Optimal energy management strategies for energy Internet via deep reinforcement learning approach Hua, Haochen; Qin, Yuchao; Hao, Chuantong Applied Energy, Vol. 239 https://doi.org/10.1016/j.apenergy.2019.01.145	journal	April 2019
A reinforcement learning framework for optimal operation and maintenance of power grids Rocchetta, R.; Bellani, L.; Compare, M. Applied Energy, Vol. 241 https://doi.org/10.1016/j.apenergy.2019.03.027	journal	May 2019
Deep reinforcement learning of energy management with continuous control strategy and traffic information for a series-parallel plug-in hybrid electric bus Wu, Yuankai; Tan, Huachun; Peng, Jiankun Applied Energy, Vol. 247 https://doi.org/10.1016/j.apenergy.2019.04.021	journal	August 2019
Energy management based on reinforcement learning with double deep Q-learning for a hybrid electric tracked vehicle Han, Xuefeng; He, Hongwen; Wu, Jingda Applied Energy, Vol. 254 https://doi.org/10.1016/j.apenergy.2019.113708	journal	November 2019
Safe deep reinforcement learning-based constrained optimal control scheme for active distribution networks Kou, Peng; Liang, Deliang; Wang, Chen Applied Energy, Vol. 264 https://doi.org/10.1016/j.apenergy.2020.114772	journal	April 2020
Towards optimal control of air handling units using deep reinforcement learning and recurrent neural network Zou, Zhengbo; Yu, Xinran; Ergan, Semiha Building and Environment, Vol. 168 https://doi.org/10.1016/j.buildenv.2019.106535	journal	January 2020
A review on buildings energy consumption information Pérez-Lombard, Luis; Ortiz, José; Pout, Christine Energy and Buildings, Vol. 40, Issue 3 https://doi.org/10.1016/j.enbuild.2007.03.007	journal	January 2008
Whole building energy model for HVAC optimal control: A practical framework based on deep reinforcement learning Zhang, Zhiang; Chong, Adrian; Pan, Yuqi Energy and Buildings, Vol. 199 https://doi.org/10.1016/j.enbuild.2019.07.029	journal	September 2019
Human-level control through deep reinforcement learning Mnih, Volodymyr; Kavukcuoglu, Koray; Silver, David Nature, Vol. 518, Issue 7540 https://doi.org/10.1038/nature14236	journal	February 2015
Mastering the game of Go with deep neural networks and tree search Silver, David; Huang, Aja; Maddison, Chris J. Nature, Vol. 529, Issue 7587 https://doi.org/10.1038/nature16961	journal	January 2016
Application of deep Q-networks for model-free optimal control balancing between different HVAC systems Ahn, Ki Uhn; Park, Cheol Soo Science and Technology for the Built Environment, Vol. 26, Issue 1 https://doi.org/10.1080/23744731.2019.1680234	journal	October 2019
DeepComfort: Energy-Efficient Thermal Comfort Control in Buildings Via Reinforcement Learning Gao, Guanyu; Li, Jie; Wen, Yonggang IEEE Internet of Things Journal, Vol. 7, Issue 9 https://doi.org/10.1109/JIOT.2020.2992117	journal	September 2020
From AlphaGo to Power System AI: What Engineers Can Learn from Solving the Most Complex Board Game Li, Fangxing; Du, Yan IEEE Power and Energy Magazine, Vol. 16, Issue 2 https://doi.org/10.1109/MPE.2017.2779554	journal	March 2018
Ancillary Services Through Demand Scheduling and Control of Commercial Buildings Lin, Yashen; Barooah, Prabir; Mathieu, Johanna L. IEEE Transactions on Power Systems, Vol. 32, Issue 1 https://doi.org/10.1109/TPWRS.2016.2557764	journal	January 2017
Transactive Control of Commercial Buildings for Demand Response Hao, He; Corbin, Charles D.; Kalsi, Karanjit IEEE Transactions on Power Systems, Vol. 32, Issue 1 https://doi.org/10.1109/TPWRS.2016.2559485	journal	January 2017
Achieving 100x Acceleration for N-1 Contingency Screening With Uncertain Scenarios Using Deep Convolutional Neural Network Du, Yan; Li, Fangxing; Li, Jiang IEEE Transactions on Power Systems, Vol. 34, Issue 4 https://doi.org/10.1109/TPWRS.2019.2914860	journal	July 2019
An Evaluation of the HVAC Load Potential for Providing Load Balancing Service Lu, Ning IEEE Transactions on Smart Grid, Vol. 3, Issue 3 https://doi.org/10.1109/TSG.2012.2183649	journal	September 2012
End-User Comfort Oriented Day-Ahead Planning for Responsive Residential HVAC Demand Aggregation Considering Weather Forecasts Erdinc, Ozan; Tascikaraoglu, Akin; Paterakis, Nikolaos G. IEEE Transactions on Smart Grid, Vol. 8, Issue 1 https://doi.org/10.1109/TSG.2016.2556619	journal	January 2017
Energy Management Considering Load Operations and Forecast Errors With Application to HVAC Systems Ma, Kai; Hu, Guoqiang; Spanos, Costas J. IEEE Transactions on Smart Grid, Vol. 9, Issue 2 https://doi.org/10.1109/TSG.2016.2558319	journal	March 2018
A Distributed Computing Platform Supporting Power System Security Knowledge Discovery Based on Online Simulation Huang, Tian-en; Guo, Qinglai; Sun, Hongbin IEEE Transactions on Smart Grid, Vol. 8, Issue 3 https://doi.org/10.1109/TSG.2016.2571442	journal	May 2017
Convolutional Neural Networks for Automatic State-Time Feature Extraction in Reinforcement Learning Applied to Residential Load Control Claessens, Bert J.; Vrancx, Peter; Ruelens, Frederik IEEE Transactions on Smart Grid, Vol. 9, Issue 4 https://doi.org/10.1109/TSG.2016.2629450	journal	July 2018
Hierarchical Control of Residential HVAC Units for Primary Frequency Regulation Wu, Xiaoyu; He, Jinghan; Xu, Yin IEEE Transactions on Smart Grid, Vol. 9, Issue 4 https://doi.org/10.1109/TSG.2017.2766880	journal	July 2018
Online Energy Management for a Sustainable Smart Home With an HVAC Load and Random Occupancy Yu, Liang; Jiang, Tao; Zou, Yulong IEEE Transactions on Smart Grid, Vol. 10, Issue 2 https://doi.org/10.1109/TSG.2017.2775209	journal	March 2019
On-Line Building Energy Optimization Using Deep Reinforcement Learning Mocanu, Elena; Mocanu, Decebal Constantin; Nguyen, Phuong H. IEEE Transactions on Smart Grid, Vol. 10, Issue 4 https://doi.org/10.1109/TSG.2018.2834219	journal	July 2019
Intelligent Multi-Microgrid Energy Management Based on Deep Neural Network and Model-Free Reinforcement Learning Du, Yan; Li, Fangxing IEEE Transactions on Smart Grid, Vol. 11, Issue 2 https://doi.org/10.1109/TSG.2019.2930299	journal	March 2020
A Scalable and Distributed Algorithm for Managing Residential Demand Response Programs Using Alternating Direction Method of Multipliers (ADMM) Kou, Xiao; Li, Fangxing; Dong, Jin IEEE Transactions on Smart Grid, Vol. 11, Issue 6 https://doi.org/10.1109/TSG.2020.2995923	journal	November 2020
Multi-Agent Deep Reinforcement Learning for HVAC Control in Commercial Buildings Yu, Liang; Sun, Yi; Xu, Zhanbo IEEE Transactions on Smart Grid, Vol. 12, Issue 1 https://doi.org/10.1109/TSG.2020.3011739	journal	January 2021
A Long-Short Term Memory Recurrent Neural Network Based Reinforcement Learning Controller for Office Heating Ventilation and Air Conditioning Systems Wang, Yuan; Velswamy, Kirubakaran; Huang, Biao Processes, Vol. 5, Issue 3, p. 46 https://doi.org/10.3390/pr5030046	journal	August 2017

Similar Records

Multi-task deep reinforcement learning for intelligent multi-zone residential HVAC control

Journal Article · Wed Nov 18 23:00:00 EST 2020 · Electric Power Systems Research · OSTI ID:1807287

Comparative analysis of model-free and model-based HVAC control for residential demand response

Conference · Mon Nov 01 00:00:00 EDT 2021 · OSTI ID:1837403

Automated Control of Transactive HVACs in Energy Distribution Systems

Journal Article · Thu Dec 03 23:00:00 EST 2020 · IEEE Transactions on Smart Grid · OSTI ID:1828730

Related Subjects

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATION
Actor-critic learning
Deep deterministic policy gradient (DDPG)
Deep reinforcement learning (deep RL)
Demand response
Multi-zone residential HVAC

Intelligent multi-zone residential HVAC control strategy based on deep reinforcement learning

Citation Formats

References (29)

Similar Records

Related Subjects