Gnu-RL: A Precocial Reinforcement Learning Solution for Building HVAC Control Using a Differentiable MPC Policy

doi:https://doi.org/10.1145/3360322.3360849

Title: Gnu-RL: A Precocial Reinforcement Learning Solution for Building HVAC Control Using a Differentiable MPC Policy

Abstract

Reinforcement learning (RL) was first demonstrated to be a feasible approach to controlling heating, ventilation, and air conditioning (HVAC) systems more than a decade ago. However, there has been limited progress towards a practical and scalable RL solution for HVAC control. While one can train an RL agent in simulation, it is not cost-effective to create a model for each thermal zone or building. Likewise, existing RL agents generally take a long time to learn and are opaque to expert interrogation, making them unattractive for real-world deployment. To tackle these challenges, we propose Gnu-RL: a novel approach that enables practical deployment of RL for HVAC control and re- quires no prior information other than historical data from exist- ing HVAC controllers. To achieve this, Gnu-RL adopts a recently- developed Differentiable Model Predictive Control (MPC) policy, which encodes domain knowledge on planning and system dynam- ics, making it both sample-efficient and interpretable. Prior to any interaction with the environment, a Gnu-RL agent is pre-trained on historical data using imitation learning, which enables it to match the behavior of the existing controller. Once it is put in charge of controlling the environment, the agent continues to improve its policy end-to-end, using amore »« less

Authors:

Chen, Bingqing ^[1]; berges, mario; Cai, Zicheng

Select or enter

Publication Date:: 2019-11-13

Research Org.:: Carnegie Mellon Univ., Pittsburgh, PA (United States)

Sponsoring Org.:: USDOE Office of Energy Efficiency and Renewable Energy (EERE), Building Technologies Office (EE-5B)

OSTI Identifier:: 1576205

Report Number(s):: DOE-CMU-EE0007682

DOE Contract Number:: EE0007682

Resource Type:: Conference

Resource Relation:: Conference: ACM BuildSys, New York, NY, Nov. 2019; Related Information: @inproceedings{Chen:2019:GPR:3360322.3360849,author = {Chen, Bingqing and Cai, Zicheng and Berg{\'e}s, Mario},title = {Gnu-RL: A Precocial Reinforcement Learning Solution for Building HVAC Control Using a Differentiable MPC Policy},booktitle = {Proceedings of the 6th ACM International Conference on Systems for Energy-Efficient Buildings, Cities, and Transportation},series = {BuildSys '19},year = {2019},isbn = {978-1-4503-7005-9},location = {New York, NY, USA},pages = {316--325},numpages = {10},url = {http://doi.acm.org/10.1145/3360322.3360849},doi = {10.1145/3360322.3360849},acmid = {3360849},publisher = {ACM},address = {New York, NY, USA},keywords = {Deep Reinforcement Learning, HVAC Control},}

Country of Publication:: United States

Language:: English

Subject:: 32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATION; HVAC: Learning; RL

Citation Formats


                    Chen, Bingqing, berges, mario, and Cai, Zicheng. Gnu-RL: A Precocial Reinforcement Learning Solution for Building HVAC Control Using a Differentiable MPC Policy.  United States: N. p., 2019. 
        Web.  doi:10.1145/3360322.3360849.

Copy to clipboard


                    Chen, Bingqing, berges, mario, & Cai, Zicheng. Gnu-RL: A Precocial Reinforcement Learning Solution for Building HVAC Control Using a Differentiable MPC Policy.  United States.  https://doi.org/10.1145/3360322.3360849

Copy to clipboard


                    Chen, Bingqing, berges, mario, and Cai, Zicheng. Wed .  
        "Gnu-RL: A Precocial Reinforcement Learning Solution for Building HVAC Control Using a Differentiable MPC Policy".  United States.  https://doi.org/10.1145/3360322.3360849.

Copy to clipboard


                    
@article{osti_1576205,

  title        = {Gnu-RL: A Precocial Reinforcement Learning Solution for Building HVAC Control Using a Differentiable MPC Policy},

  author       = {Chen, Bingqing and berges, mario and Cai, Zicheng},

  abstractNote = {Reinforcement learning (RL) was first demonstrated to be a feasible approach to controlling heating, ventilation, and air conditioning (HVAC) systems more than a decade ago. However, there has been limited progress towards a practical and scalable RL solution for HVAC control. While one can train an RL agent in simulation, it is not cost-effective to create a model for each thermal zone or building. Likewise, existing RL agents generally take a long time to learn and are opaque to expert interrogation, making them unattractive for real-world deployment. To tackle these challenges, we propose Gnu-RL: a novel approach that enables practical deployment of RL for HVAC control and re- quires no prior information other than historical data from exist- ing HVAC controllers. To achieve this, Gnu-RL adopts a recently- developed Differentiable Model Predictive Control (MPC) policy, which encodes domain knowledge on planning and system dynam- ics, making it both sample-efficient and interpretable. Prior to any interaction with the environment, a Gnu-RL agent is pre-trained on historical data using imitation learning, which enables it to match the behavior of the existing controller. Once it is put in charge of controlling the environment, the agent continues to improve its policy end-to-end, using a policy gradient algorithm. We evaluate Gnu-RL on both an EnergyPlus model and a real- world testbed. In both experiments, our agents were directly de- ployed in the environment after offline pre-training on expert demonstration. In the simulation experiment, our approach saved 6.6% energy compared to the best published RL result for the same environment, while maintaining a higher level of occupant comfort. Next, Gnu-RL was deployed to control the HVAC of a real-world conference room for a three-week period. Our results show that Gnu-RL saved 16.7% of cooling demand compared to the existing controller and tracked temperature set-point better.},

  doi          = {10.1145/3360322.3360849},

  url          = {https://www.osti.gov/biblio/1576205},
  journal      = {},
number       = ,

  volume       = ,

  place        = {United States},

  year         = {2019},

  month        = {11}

}

Copy to clipboard

Conference:

https://doi.org/10.1145/3360322.3360849

Other availability

Please see Document Availability for additional information on obtaining the full-text document. Library patrons may search WorldCat to identify libraries that hold this conference proceeding.

Save / Share:

Export Metadata

Save to My Library

Works referenced in this record:

Reducing Transient and Steady State Electricity Consumption in HVAC Using Learning-Based Model-Predictive Control
journal, January 2012

Aswani, A.; Master, N.; Taneja, J.
Proceedings of the IEEE, Vol. 100, Issue 1
https://doi.org/10.1109/JPROC.2011.2161242

Similar records in OSTI.GOV collections:

Evaluating the Adaptability of Reinforcement Learning Based HVAC Control for Residential Houses

Journal Article Kurte, Kuldeep ; Munk, Jeffrey ; Kotevska, Olivera ; ... - Sustainability (Basel)

Intelligent Heating, Ventilation, and Air Conditioning (HVAC) control using deep reinforcement learning (DRL) has recently gained a lot of attention due to its ability to optimally control the complex behavior of the HVAC system. However, more exploration is needed on understanding the adaptability challenges that the DRL agent could face during the deployment phase. Using online learning for such applications is not realistic due to the long learning period and likely poor comfort control during the learning process. Alternatively, DRL can be pre-trained using a building model prior to deployment. However, developing an accurate building model for every house andmore »« less
https://doi.org/10.3390/su12187727

Full Text Available
VOLTTRON COMPATIBLE WHOLE BUILDING ROOTFAULT DETECTION AND DIAGNOSIS

Technical Report Wen, Jin ; Chen, Yimin

Buildings consume more than 40% of primary energy in the U.S. Malfunctioning sensors, components, and control systems, as well as degrading components in Heating, Ventilating and Air-conditioning (HVAC) and lighting systems are main reasons for energy waste and unsatisfactory indoor environment. Field studies have shown that an energy saving of 5-30% of total building energy consumption and improved indoor air quality can be achieved by simply applying automated fault detection and diagnosis (AFDD), followed by corrections even if these are not done in real time. Extensive research has been made on the development of component level AFDD tools in themore »« less
https://doi.org/10.2172/1524639

Full Text Available
RL-HEMS: Reinforcement Learning Based Home Energy Management System for HVAC Energy Optimization

Conference Kotevska, Olivera ; Kurte, Kuldeep ; Munk, Jeffrey ; ...

Heating, ventilation, and air conditioning (HVAC) is one of the major energy consumers in the residential sector. It is important to be able to monitor and control the energy consumed to provide utility services such as load shaping while satisfying the comfort and economic constraints of the homeowner. The objective of this work is to create the optimal schedule for HVAC operation to reduce the cost while satisfying the home-owner and equipment’s constraints using a model-free Reinforcement Learning (RL)-based optimization. The specific goal is to find the right balance between reducing energy cost, consumption, and customer comfort level. Our researchmore »« less
An Edge-Cloud Integrated Solution for Buildings Demand Response Using Reinforcement Learning

Journal Article Zhang, Xiangyu ; Biagioni, Dave ; Cai, Mengmeng ; ... - IEEE Transactions on Smart Grid

Buildings, as major energy consumers, can provide great untapped demand response (DR) resources for grid services. However, their participation remains low in real-life. One major impediment for popularizing DR in buildings is the lack of cost-effective automation systems that can be widely adopted. Existing optimization-based smart building control algorithms suffer from high costs on both building-specific modeling and on-demand computing resources. To tackle these issues, this paper proposes a cost-effective edge-cloud integrated solution using reinforcement learning (RL). Beside RL’s ability to solve sequential optimal decision-making problems, its adaptability to easy-to-obtain building models and the off-line learning feature are likely tomore »« less
https://doi.org/10.1109/tsg.2020.3014055
Policy, Permit, Perform: Using City Benchmarking Data and Building Construction Permit History to Identify Energy Performance Improvements

Technical Report Stevens, Casey ; Tyler, Matt ; Lee, Allen ; ...

The City of Portland (City) and Multnomah County 2015 Climate Action Plan targets a 40 percent reduction in carbon emissions below 1990 levels by 2030. To reach this goal, the plan includes a key objective to reduce the total energy use of existing buildings by 25 percent. Buildings are responsible for one-half of carbon emissions in Portland, Oregon, U.S., and improving their performance is critical to achieving the City’s climate goals. As part of its climate action planning, the City passed an ordinance on April 22, 2015, requiring commercial buildings 20,000 ft2 and larger to benchmark and disclose annual energymore »« less
https://doi.org/10.2172/1477793

Full Text Available

Similar Records

Title: Gnu-RL: A Precocial Reinforcement Learning Solution for Building HVAC Control Using a Differentiable MPC Policy

Abstract

Citation Formats

Reducing Transient and Steady State Electricity Consumption in HVAC Using Learning-Based Model-Predictive Control journal, January 2012

Reducing Transient and Steady State Electricity Consumption in HVAC Using Learning-Based Model-Predictive Control
journal, January 2012