Two-Stage Reinforcement Learning Policy Search for Grid-Interactive Building Control
This paper develops an intelligent grid-interactive building controller, which optimizes building operation during both normal hours and demand response (DR) events. To avoid costly on-demand computation and to adapt to non-linear building models, the controller utilizes reinforcement learning (RL) and makes real-time decisions based on a near-optimal control policy. Learning such a policy typically amounts to solving a hard non-convex optimization problem. We propose to address this problem with a novel global-local policy search method. In the first stage, an RL algorithm based on zero-order gradient estimation is leveraged to search for the optimal policy globally, due to its scalability and the potential to escape some poor performing local optima. The obtained policy is then fine-tuned locally to bring the first-stage solution closer to that of the original unsmoothed problem. Experiments on a simulated five-zone commercial building demonstrate the advantages of the proposed method over existing learning approaches. They also show that the learned control policy outperforms a pragmatic linear model predictive controller (MPC) and approaches the performance of an oracle MPC in testing scenarios. Using a state-of-the-art advanced computing system, we demonstrate that the controller can be learned and deployed within hours of training.
- Research Organization:
- National Renewable Energy Laboratory (NREL), Golden, CO (United States)
- Sponsoring Organization:
- USDOE Office of Energy Efficiency and Renewable Energy (EERE), Energy Efficiency Office. Building Technologies Office
- Grant/Contract Number:
- AC36-08GO28308
- OSTI ID:
- 1841140
- Report Number(s):
- NREL/JA-2C00-79559; MainId:35780; UUID:ae6b3461-9d61-4782-8331-aa0d27f35eab; MainAdminID:63638
- Journal Information:
- IEEE Transactions on Smart Grid, Journal Name: IEEE Transactions on Smart Grid Journal Issue: 3 Vol. 13; ISSN 1949-3053
- Publisher:
- IEEECopyright Statement
- Country of Publication:
- United States
- Language:
- English
Similar Records
Grid-Interactive Multi-Zone Building Control Using Reinforcement Learning with Global-Local Policy Search: Preprint
Grid-Interactive Multi-Zone Building Control Using Reinforcement Learning with Global-Local Policy Search