skip to main content
OSTI.GOV title logo U.S. Department of Energy
Office of Scientific and Technical Information

Title: Online transfer learning strategy for enhancing the scalability and deployment of deep reinforcement learning control in smart buildings

Journal Article · · Applied Energy
ORCiD logo [1];  [1]; ORCiD logo [2]; ORCiD logo [1]
  1. Politecnico di Torino (Italy)
  2. Lawrence Berkeley National Laboratory (LBNL), Berkeley, CA (United States)
+ Show Author Affiliations

In recent years, advanced control strategies based on Deep Reinforcement Learning (DRL) proved to be effective in optimizing the management of integrated energy systems in buildings, reducing energy costs and improving indoor comfort conditions when compared to traditional reactive controllers. However, the scalability and implementation of DRL controllers are still limited since they require a considerable amount of time before converging to a near-optimal solution. This issue is currently addressed in literature through the offline pre-training of the DRL agent. However this solution results in two main critical issues: (1) the need to develop a building surrogate model to perform the training task, and (2) the need to perform a fine-tuning process over several training episodes to obtain a near-optimal control policy. In this context, this paper introduces an Online Transfer Learning (OTL) strategy that exploits two knowledge-sharing techniques, weight-initialization and imitation learning, to transfer a DRL control policy from a source office building to various target buildings in a simulation environment coupling EnergyPlus and Python. A DRL controller based on discrete Soft Actor–Critic (SAC) is trained on the source building to manage the operation of a cooling system consisting of a chiller and a thermal storage. Several target buildings are defined to benchmark the performance of the OTL strategy with that of a Rule-Based Controller (RBC) and two DRL-based control strategies, deployed in offline and online fashion. The strategy adopted for OTL emulates the real world implementation with a simulation process by implementing the transferred DRL agent for a single episode in the target buildings. Target buildings have the same geometrical features and are served by the same energy system as the source building, but differ in terms of weather conditions, electricity price schedules, occupancy patterns, and building envelope efficiency levels. The results show that the OTL strategy can reduce the cumulated sum of temperature violations on average by 50% and 80% respectively when compared to RBC and online DRL while enhancing the energy system operation with electricity cost savings ranging between 20% and 40%. Furthermore, the OTL agent performs slightly worse than the offline DRL controller but it does not require any modeling effort and can be implemented directly on target buildings emulating a real-world implementation.

Research Organization:
Lawrence Berkeley National Laboratory (LBNL), Berkeley, CA (United States)
Sponsoring Organization:
USDOE Office of Energy Efficiency and Renewable Energy (EERE), Energy Efficiency Office. Building Technologies Office
Grant/Contract Number:
AC02-05CH11231
OSTI ID:
1984646
Journal Information:
Applied Energy, Vol. 333; ISSN 0306-2619
Publisher:
ElsevierCopyright Statement
Country of Publication:
United States
Language:
English

References (51)

Transfer learning in demand response: A review of algorithms for data-efficient modelling and control journal January 2022
Transfer Learning Applied to Reinforcement Learning-Based HVAC Control journal April 2020
Dynamic Programming journal July 1966
Forecasting Energy Consumption of Wastewater Treatment Plants with a Transfer Learning Approach for Sustainable Cities journal May 2021
Sharing is caring: An extensive analysis of parameter-based transfer learning for the prediction of building thermal dynamics journal December 2022
Energy-Efficiency Requirements for Residential Building Envelopes in Cold-Climate Regions journal March 2021
A hybrid deep transfer learning strategy for short term cross-building energy prediction journal January 2021
A Survey of Control Technologies in the Building Automation Industry journal January 2005
Comparison of online and offline deep reinforcement learning with model predictive control for thermal energy management journal March 2022
Using transfer learning for smart building management system journal December 2019
Mastering the game of Go with deep neural networks and tree search journal January 2016
On the Establishment of Climatic Zones in Europe with Regard to the Energy Performance of Buildings journal December 2011
Data-centric or algorithm-centric: Exploiting the performance of transfer learning for improving building energy predictions in data-scarce context journal February 2022
All you need to know about model predictive control for buildings journal January 2020
Deep reinforcement learning to optimise indoor temperature control and heating energy consumption in buildings journal October 2020
A Procedure for Automating Energy Analyses in the BIM Context Exploiting Artificial Neural Networks and Transfer Learning Technique journal May 2021
Optimal sizing design and operation of electrical and thermal energy storage systems in smart buildings journal April 2020
A Long-Short Term Memory Recurrent Neural Network Based Reinforcement Learning Controller for Office Heating Ventilation and Air Conditioning Systems journal August 2017
A survey of robot learning from demonstration journal May 2009
Optimal Scheduling of Microgrid Based on Deep Deterministic Policy Gradient and Transfer Learning journal January 2021
Transfer learning applied to DRL-Based heat pump control to leverage microgrid energy efficiency journal August 2021
Fusing TensorFlow with building energy simulation for intelligent energy management in smart cities journal February 2019
Intelligent multi-zone residential HVAC control strategy based on deep reinforcement learning journal January 2021
Generalized online transfer learning for climate control in residential buildings journal March 2017
Online Implementation of a Soft Actor-Critic Agent to Enhance Indoor Temperature Control and Energy Efficiency in Buildings journal February 2021
Reinforcement learning for building controls: The opportunities and challenges journal July 2020
A concept of compromise solutions and the method of the displaced ideal journal December 1974
Human-level control through deep reinforcement learning journal February 2015
Model Predictive Control (MPC) for Enhancing Building and HVAC System Energy Efficiency: Problem Formulation, Applications and Opportunities journal March 2018
Whole building energy model for HVAC optimal control: A practical framework based on deep reinforcement learning journal September 2019
Online Transfer Learning journal November 2014
Reinforcement learning for optimal control of low exergy buildings journal October 2015
Energy optimization associated with thermal comfort and indoor air control via a deep reinforcement learning algorithm journal May 2019
Next-generation energy systems for sustainable smart cities: Roles of transfer learning journal October 2022
A comparative review of heating systems in EU countries, based on efficiency and fuel cost journal July 2018
EnergyPlus: creating a new-generation building energy simulation program journal April 2001
A Survey on Transfer Learning journal October 2010
Transfer learning with deep neural networks for model predictive control of HVAC and natural ventilation in smart buildings journal May 2020
Development of a Predictive Optimal Controller for Thermal Energy Storage Systems journal July 1997
A transferable energy management strategy for hybrid electric vehicles via dueling deep deterministic policy gradient journal September 2022
Towards optimal control of air handling units using deep reinforcement learning and recurrent neural network journal January 2020
A Comparative Analysis of the International Regulation of Thermal Properties in Building Envelope journal October 2019
A predictive and adaptive control strategy to optimize the management of integrated energy systems in buildings journal November 2022
Formulation of a model predictive control algorithm to enhance the performance of a latent heat solar thermal system journal October 2018
Transferable Reinforcement Learning for Smart Homes
  • Zhang, Xiangyu; Jin, Xin; Tripp, Charles
  • BuildSys '20: The 7th ACM International Conference on Systems for Energy-Efficient Buildings, Cities, and Transportation, Proceedings of the 1st International Workshop on Reinforcement Learning for Energy Management in Buildings & Cities https://doi.org/10.1145/3427773.3427865
conference November 2020
Use of model predictive control and weather forecasts for energy efficient building climate control journal February 2012
Data-driven district energy management with surrogate models and deep reinforcement learning journal December 2021
Coordinated energy management for a cluster of buildings through deep reinforcement learning journal August 2021
Deep Reinforcement Learning-Based Image Captioning with Embedding Reward conference July 2017
Transfer learning for smart buildings: A critical review of algorithms, applications, and future perspectives journal February 2022
Simulation-Based Evaluation and Optimization of Control Strategies in Buildings journal December 2018

Similar Records

Related Subjects

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATION
Online transfer learning
Homogeneous transfer learning
Intra-agent transfer learning
Building adaptive control
Deep reinforcement learning
Energy efficiency