Data-driven reinforcement learning–based real-time energy management system for plug-in hybrid electric vehicles

Qi, Xuewei; Wu, Guoyuan; Boriboonsomsin, Kanok; Barth, Matthew J.; Gonder, Jeffrey

doi:10.3141/2572-01

Title: Data-driven reinforcement learning–based real-time energy management system for plug-in hybrid electric vehicles

Abstract

Plug-in hybrid electric vehicles (PHEVs) show great promise in reducing transportation-related fossil fuel consumption and greenhouse gas emissions. Designing an efficient energy management system (EMS) for PHEVs to achieve better fuel economy has been an active research topic for decades. Most of the advanced systems rely either on a priori knowledge of future driving conditions to achieve the optimal but not real-time solution (e.g., using a dynamic programming strategy) or on only current driving situations to achieve a real-time but nonoptimal solution (e.g., rule-based strategy). This paper proposes a reinforcement learning–based real-time EMS for PHEVs to address the trade-off between real-time performance and optimal energy savings. The proposed model can optimize the power-split control in real time while learning the optimal decisions from historical driving cycles. Here, a case study on a real-world commute trip shows that about a 12% fuel saving can be achieved without considering charging opportunities; further, an 8% fuel saving can be achieved when charging opportunities are considered, compared with the standard binary mode control strategy.

Authors:

Qi, Xuewei ^[1]; Wu, Guoyuan ^[1]; Boriboonsomsin, Kanok ^[1]; Barth, Matthew J. ^[1]; Gonder, Jeffrey ^[2]

Univ. of California, Riverside, CA (United States)
National Renewable Energy Lab. (NREL), Golden, CO (United States)

Publication Date:: Fri Jan 01 00:00:00 EST 2016

Research Org.:: National Renewable Energy Laboratory (NREL), Golden, CO (United States)

Sponsoring Org.:: U.S. Department of Transportation

OSTI Identifier:: 1239892

Report Number(s):: NREL/JA-5400-65413
Journal ID: ISSN 0361-1981

Grant/Contract Number:: AC36-08GO28308

Resource Type:: Accepted Manuscript

Journal Name:: Transportation Research Record: Journal of the Transportation Research Board

Additional Journal Information:: Journal Volume: 2572; Journal ID: ISSN 0361-1981

Publisher:: National Academy of Sciences, Engineering and Medicine

Country of Publication:: United States

Language:: English

Subject:: 33 ADVANCED PROPULSION SYSTEMS; plug-in hybrid electric vehicles (PHEV); energy management system (EMS); approximate dynamic programming; reinforcement learning (RL)

Citation Formats


                    Qi, Xuewei, Wu, Guoyuan, Boriboonsomsin, Kanok, Barth, Matthew J., and Gonder, Jeffrey. Data-driven reinforcement learning–based real-time energy management system for plug-in hybrid electric vehicles.  United States: N. p., 2016. 
Web.  doi:10.3141/2572-01.

Copy to clipboard


                    Qi, Xuewei, Wu, Guoyuan, Boriboonsomsin, Kanok, Barth, Matthew J., & Gonder, Jeffrey. Data-driven reinforcement learning–based real-time energy management system for plug-in hybrid electric vehicles.  United States.  https://doi.org/10.3141/2572-01

Copy to clipboard


                    Qi, Xuewei, Wu, Guoyuan, Boriboonsomsin, Kanok, Barth, Matthew J., and Gonder, Jeffrey. Fri .  
"Data-driven reinforcement learning–based real-time energy management system for plug-in hybrid electric vehicles".  United States.  https://doi.org/10.3141/2572-01.  https://www.osti.gov/servlets/purl/1239892.

Copy to clipboard


                    
@article{osti_1239892,

  title        = {Data-driven reinforcement learning–based real-time energy management system for plug-in hybrid electric vehicles},

  author       = {Qi, Xuewei and Wu, Guoyuan and Boriboonsomsin, Kanok and Barth, Matthew J. and Gonder, Jeffrey},

  abstractNote = {Plug-in hybrid electric vehicles (PHEVs) show great promise in reducing transportation-related fossil fuel consumption and greenhouse gas emissions. Designing an efficient energy management system (EMS) for PHEVs to achieve better fuel economy has been an active research topic for decades. Most of the advanced systems rely either on a priori knowledge of future driving conditions to achieve the optimal but not real-time solution (e.g., using a dynamic programming strategy) or on only current driving situations to achieve a real-time but nonoptimal solution (e.g., rule-based strategy). This paper proposes a reinforcement learning–based real-time EMS for PHEVs to address the trade-off between real-time performance and optimal energy savings. The proposed model can optimize the power-split control in real time while learning the optimal decisions from historical driving cycles. Here, a case study on a real-world commute trip shows that about a 12% fuel saving can be achieved without considering charging opportunities; further, an 8% fuel saving can be achieved when charging opportunities are considered, compared with the standard binary mode control strategy.},

  doi          = {10.3141/2572-01},

  journal      = {Transportation Research Record: Journal of the Transportation Research Board},

  number       = ,

  volume       = 2572,

  place        = {United States},

  year         = {Fri Jan 01 00:00:00 EST 2016},

  month        = {Fri Jan 01 00:00:00 EST 2016}

}

Copy to clipboard

Journal Article:

Free Publicly Available Full Text

Accepted Manuscript (DOE)

Publisher's Version of Record

https://doi.org/10.3141/2572-01

Other availability

Search WorldCat to find libraries that may hold this journal

Citation Metrics:

Cited by: 81 works

Citation information provided by
Web of Science

Save / Share:

Export Metadata

Save to My Library

Works referenced in this record:

Approximate Dynamic Programming: Solving the Curses of Dimensionality
book, August 2011

Powell, Warren B.
Wiley Series in Probability and Statistics
DOI: 10.1002/9781118029176

Adaptive traffic signal control using approximate dynamic programming
journal, October 2009

Cai, Chen; Wong, Chi Kwong; Heydecker, Benjamin G.
Transportation Research Part C: Emerging Technologies, Vol. 17, Issue 5
DOI: 10.1016/j.trc.2009.04.005

Modeling and Control of a Power-Split Hybrid Vehicle
journal, November 2008

Jinming Liu,
IEEE Transactions on Control Systems Technology, Vol. 16, Issue 6
DOI: 10.1109/TCST.2008.919447

Commuter Route Optimized Energy Management of Hybrid Electric Vehicles
journal, June 2014

Larsson, Viktor; Johannesson Mardh, Lars; Egardt, Bo
IEEE Transactions on Intelligent Transportation Systems, Vol. 15, Issue 3
DOI: 10.1109/TITS.2013.2294723

Comparative Study on Different Energy Management Strategies for Plug-In Hybrid Electric Vehicles
journal, October 2013

Wang, Ximing; He, Hongwen; Sun, Fengchun
Energies, Vol. 6, Issue 11
DOI: 10.3390/en6115656

A Supervisory Control Strategy for Plug-In Hybrid Electric Vehicles Based on Energy Demand Prediction and Route Preview
journal, May 2015

Tianheng, Feng; Lin, Yang; Qing, Gu
IEEE Transactions on Vehicular Technology, Vol. 64, Issue 5
DOI: 10.1109/TVT.2014.2336378

A Real Time Energy Management Strategy for Plug-in Hybrid Electric Vehicles based on Optimal Control Theory
journal, January 2014

Tribioli, Laura; Barbieri, Michele; Capata, Roberto
Energy Procedia, Vol. 45
DOI: 10.1016/j.egypro.2014.01.100

Energy management of a power-split plug-in hybrid electric vehicle based on genetic algorithm and quadratic programming
journal, February 2014

Chen, Zheng; Mi, Chris Chunting; Xiong, Rui
Journal of Power Sources, Vol. 248
DOI: 10.1016/j.jpowsour.2013.09.085

Energy management of plug-in hybrid electric vehicles with unknown trip length
journal, February 2015

Hou, Cong; Xu, Liangfei; Wang, Hewu
Journal of the Franklin Institute, Vol. 352, Issue 2
DOI: 10.1016/j.jfranklin.2014.07.009

Development and Evaluation of an Intelligent Energy-Management Strategy for Plug-in Hybrid Electric Vehicles
journal, June 2014

Wu, Guoyuan; Boriboonsomsin, Kanok; Barth, Matthew J.
IEEE Transactions on Intelligent Transportation Systems, Vol. 15, Issue 3
DOI: 10.1109/TITS.2013.2294342

Trip-Oriented Energy Management Control Strategy for Plug-In Hybrid Electric Vehicles
journal, July 2014

Yu, Hai; Kuang, Mong; McGee, Ryan
IEEE Transactions on Control Systems Technology, Vol. 22, Issue 4, p. 1323-1336
DOI: 10.1109/TCST.2013.2278684

Fuzzy-based blended control for the energy management of a parallel plug-in hybrid electric vehicle
journal, February 2015

Dubois, Maxime R.; Desrochers, Alain; Denis, Nicolas
IET Intelligent Transport Systems, Vol. 9, Issue 1
DOI: 10.1049/iet-its.2014.0075

Fuzzy energy management strategy for plug-in hev based on driving cycle modeling
conference, July 2014

Wu, Jian
2014 33rd Chinese Control Conference (CCC), Proceedings of the 33rd Chinese Control Conference
DOI: 10.1109/ChiCC.2014.6895690

Analysis of energy management strategies in plug-in hybrid electric vehicles: Application to the GM Chevrolet Volt
conference, June 2013

Tribioli, Laura; Onori, Simona
2013 American Control Conference (ACC)
DOI: 10.1109/ACC.2013.6580774

Trip Based Near Globally Optimal Power Management of Plug-in Hybrid Electric Vehicles Using Gas-Kinetic Traffic Flow Model
journal, January 2008

Gong, Qiuming; Li, Yaoyu; Peng, Zhong-Ren
IFAC Proceedings Volumes, Vol. 41, Issue 2
DOI: 10.3182/20080706-5-KR-1001.00785

Evolutionary algorithm based on-line PHEV energy management system with self-adaptive SOC control
conference, June 2015

Qi, Xuewei; Wu, Guoyuan; Boriboonsomsin, Kanok
2015 IEEE Intelligent Vehicles Symposium (IV)
DOI: 10.1109/IVS.2015.7225722

Trip-oriented Energy Management Control strategy for plug-in hybrid electric vehicles
conference, December 2011

Yu, Hai; Kuang, Ming; McGee, Ryan
2011 50th IEEE Conference on Decision and Control and European Control Conference (CDC-ECC 2011)
DOI: 10.1109/cdc.2011.6160609

Approximate Dynamic Programming: Solving the Curses of Dimensionality
book, January 2007

Powell, Warren B.
DOI: 10.1002/9780470182963

Works referencing / citing this record:

Energy Management Strategy for a Hybrid Electric Vehicle Based on Deep Reinforcement Learning
journal, January 2018

Hu, Yue; Li, Weimin; Xu, Kun
Applied Sciences, Vol. 8, Issue 2
DOI: 10.3390/app8020187

Similar Records in DOE PAGES and OSTI.GOV collections:

Plug-in hybrid electric vehicles : How does one determine their potential for reducing U.S. oil dependence?

Journal Article Vyas, A ; Santini, D ; Duoba, M ; ... - WEVA

Estimation of the potential of plug-in hybrid electric vehicles (PHEV's) ability to reduce U.S. gasoline use is difficult and complex. Although techniques have been proposed to estimate the vehicle kilometers of travel (VKT) that can be electrified, these methods may be inadequate and/or inappropriate for early market introduction circumstances. Factors that must be considered with respect to the PHEV itself include (1) kWh battery storage capability; (2) kWh/km depletion rate of the vehicle (3) liters/km use of gasoline (4) average daily kilometers driven (5) annual share of trips exceeding the battery depletion distance (6) driving cycle(s) (7) charger location [i.e.more »« less
Cost-Effective and Ecofriendly Plug-In Hybrid Electric Vehicle Charging Management

Journal Article Kontou, Eleftheria ; Yin, Yafeng ; Ge, Ying-en - Transportation Research Record: Journal of the Transportation Research Board

In this study we explore two charging management schemes for plug-in hybrid electric vehicles (PHEVs). The PHEV drivers and the government were stakeholders who might have preferred different charging control strategies. For the former, a proposed controlled charging scheme minimized the operational cost during PHEV charge-depleting and sustaining modes. For the latter, the research minimized monetized carbon dioxide emissions from electricity generation for the PHEVs charging, as well as tailpipe emissions for the portion of PHEV trips fueled by gasoline. Hourly driving patterns and electricity data were leveraged. Both were representative of each of the eight North American Electric Reliabilitymore »« less
Cited by 18
https://doi.org/10.3141/2628-10

Full Text Available
Using Global Positioning System Travel Data to Assess Real-World Energy Use of Plug-In Hybrid Electric Vehicles

Journal Article Gonder, J ; Markel, T ; Thornton, M ; ... - Transportation Research Record: Journal of the Transportation Research Board

Plug-in hybrid electric vehicles (PHEVs) have received considerable recent attention for their potential to reduce petroleum consumption significantly and quickly in the transportation sector. Analysis to aid the design of such vehicles and predict their real-world performance and fuel displacement must consider the driving patterns the vehicles will typically encounter. This paper goes beyond consideration of standardized certification cycless by leveraging state-of-the-art travel survey techniques that use Global Positioning System (GPS) technology to obtain a large set of real-world drive cycles from the surveyed vehicle fleet. This study specifically extracts 24-h, second-by-second driving profiles from a set of 227 GPS-instrumentedmore »« less
https://doi.org/10.3141/2017-04
High-dimensional Data-driven Energy optimization for Multi-Modal Transit Agencies (HD-EMMA) (Final Technical Report)

Technical Report Pugliese, Philip ; Dubey, Abhishek ; Laszka, Aron ; ...

Public bus transit services in the U.S. are responsible for at least 19.7 million metric tons of CO₂ emission annually. Electric vehicles (EVs) can have a much lower environmental impact than comparable internal combustion engine vehicles (ICEVs), especially in urban areas. Unfortunately, EVs are also much more expensive than ICEVs. As a result, many public transit agencies can afford only mixed fleets of transit vehicles, consisting of EVs, hybrids (HEVs), and ICEVs. Transit agencies that operate such mixed fleets of vehicles face a challenging optimization problem: these agencies need to decide which vehicles are assigned to serving which transit trips. Since the advantage of EVs over ICEVs varies depending on the route and time of day (e.g., the benefit of EVs is higher in slower traffic with frequent stops and lower on highways), the assignment can have a significant effect on energy use and, hence, environmental impact. Through this project, we have developed reference data about energy collections and constructed a set of machine learning models that can accurately predict the energy consumption for the whole fleet at the level of each trip. We have used these models to develop a scheduling and assignment strategy that can rotate the different vehicle types across the transit agencies’ routes. The optimization algorithm ensures that the vehicles are matched to trips considering weather patterns, expected congestion, and road gradients to minimize the overall energy usage. We list the key observations from our project for other practitioners below. Details are available in the report, and the list of source code and our publications are included in the appendix. 1. We have demonstrated the feasibility of collecting, merging and analyzing large volumes of high-resolution real-world telemetry data from a mixed vehicle fleet. To mitigate the inherent noise of the recorded GPS points, the team developed an algorithm that filters data and maps the points onto a street. The algorithm considers previous and subsequent location measurements and different characteristics of nearby streets to determine how likely the vehicle travels on them. Then, the team segmented the time series into disjoint contiguous samples based on adjacent road segments and repeated the outlier detection and removal. For each data point, the team added features corresponding to elevation changes within the samples, weather features, such as temperature, and traffic data, such as speed ratio between actual speed and free-flow speed. 2. We have developed two forms of machine learning models that be used to understand and analyze the energy operations of a mixed vehicle transit fleet. The micro prediction model provides estimates of instantaneous energy prediction for all types of buses (diesel, hybrid, and electric). Such a model is important in evaluating the energy impacts of real-time bus operation strategies, but it is challenging due to diversified driving cycles of transit buses. The model can help the drivers understand the impact of their driving behaviors and short-term congestions. The macro prediction models estimate average energy consumption across the whole trip considering the features: distance traveled, various road-type features, elevation change, day of the week, time of day, various weather features (temperature, humidity, etc.), and traffic features (speed ratio and jam factor). 3. We have demonstrated that it is possible to transfer the machine learning models we have developed in this project to other teams and cities by using inductive transfer learning. We also showed that the performance of the macro energy prediction models can be improved using a multi-task learning approach where the learning parameters are shared between the models being developed for different vehicle types. The advantage of this approach is improved learning performance as the models can exploit common spatio-temporal and environmental characteristics. 4. Finally, we have developed trip and vehicle assignment and scheduling algorithms that use the energy prediction models and develop a trip to vehicle type (diesel, electric, hybrid) assignment for the whole operation to reduce overall emissions and cost. We have shown through simulations that the proposed algorithms can savemore »« less
https://doi.org/10.2172/1834379

Full Text Available
Multi-modal Energy-optimal Trip Scheduling in Real-time (METS-R) for Transportation Hubs (Final Report)

Technical Report Ukkusuri, Satish V. ; Sobolevsky, Stanislav ; Kulkarni, Milind ; ...

This report summarizes the work performed under the award number EE0008524. The project develops the Multi-modal Energy-optimal Trip Scheduling in Real-time (METS-R) platform as the next-generation transportation solution based on autonomous electric vehicles (AEV) serving passenger trips from and to urban transportation hubs, to substantially reduce transportation energy consumption. Extensive data collection and analyses were first conducted to understand the demand patterns and energy consumption of hub-based on-road trips. Then, a data-driven framework that consists of an analytical module and a simulation module was proposed. For the analytical module, five planning + operation tools were developed to support the planningmore »« less
https://doi.org/10.2172/1859675

Full Text Available

Similar Records

Title: Data-driven reinforcement learning–based real-time energy management system for plug-in hybrid electric vehicles

Abstract

Citation Formats

Approximate Dynamic Programming: Solving the Curses of Dimensionality book, August 2011

Adaptive traffic signal control using approximate dynamic programming journal, October 2009

Modeling and Control of a Power-Split Hybrid Vehicle journal, November 2008

Commuter Route Optimized Energy Management of Hybrid Electric Vehicles journal, June 2014

Comparative Study on Different Energy Management Strategies for Plug-In Hybrid Electric Vehicles journal, October 2013

A Supervisory Control Strategy for Plug-In Hybrid Electric Vehicles Based on Energy Demand Prediction and Route Preview journal, May 2015

A Real Time Energy Management Strategy for Plug-in Hybrid Electric Vehicles based on Optimal Control Theory journal, January 2014

Energy management of a power-split plug-in hybrid electric vehicle based on genetic algorithm and quadratic programming journal, February 2014

Energy management of plug-in hybrid electric vehicles with unknown trip length journal, February 2015

Development and Evaluation of an Intelligent Energy-Management Strategy for Plug-in Hybrid Electric Vehicles journal, June 2014

Trip-Oriented Energy Management Control Strategy for Plug-In Hybrid Electric Vehicles journal, July 2014

Fuzzy-based blended control for the energy management of a parallel plug-in hybrid electric vehicle journal, February 2015

Fuzzy energy management strategy for plug-in hev based on driving cycle modeling conference, July 2014

Analysis of energy management strategies in plug-in hybrid electric vehicles: Application to the GM Chevrolet Volt conference, June 2013

Trip Based Near Globally Optimal Power Management of Plug-in Hybrid Electric Vehicles Using Gas-Kinetic Traffic Flow Model journal, January 2008

Evolutionary algorithm based on-line PHEV energy management system with self-adaptive SOC control conference, June 2015

Trip-oriented Energy Management Control strategy for plug-in hybrid electric vehicles conference, December 2011

Approximate Dynamic Programming: Solving the Curses of Dimensionality book, January 2007

Energy Management Strategy for a Hybrid Electric Vehicle Based on Deep Reinforcement Learning journal, January 2018

Approximate Dynamic Programming: Solving the Curses of Dimensionality
book, August 2011

Adaptive traffic signal control using approximate dynamic programming
journal, October 2009

Modeling and Control of a Power-Split Hybrid Vehicle
journal, November 2008

Commuter Route Optimized Energy Management of Hybrid Electric Vehicles
journal, June 2014

Comparative Study on Different Energy Management Strategies for Plug-In Hybrid Electric Vehicles
journal, October 2013

A Supervisory Control Strategy for Plug-In Hybrid Electric Vehicles Based on Energy Demand Prediction and Route Preview
journal, May 2015

A Real Time Energy Management Strategy for Plug-in Hybrid Electric Vehicles based on Optimal Control Theory
journal, January 2014

Energy management of a power-split plug-in hybrid electric vehicle based on genetic algorithm and quadratic programming
journal, February 2014

Energy management of plug-in hybrid electric vehicles with unknown trip length
journal, February 2015

Development and Evaluation of an Intelligent Energy-Management Strategy for Plug-in Hybrid Electric Vehicles
journal, June 2014

Trip-Oriented Energy Management Control Strategy for Plug-In Hybrid Electric Vehicles
journal, July 2014

Fuzzy-based blended control for the energy management of a parallel plug-in hybrid electric vehicle
journal, February 2015

Fuzzy energy management strategy for plug-in hev based on driving cycle modeling
conference, July 2014

Analysis of energy management strategies in plug-in hybrid electric vehicles: Application to the GM Chevrolet Volt
conference, June 2013

Trip Based Near Globally Optimal Power Management of Plug-in Hybrid Electric Vehicles Using Gas-Kinetic Traffic Flow Model
journal, January 2008

Evolutionary algorithm based on-line PHEV energy management system with self-adaptive SOC control
conference, June 2015

Trip-oriented Energy Management Control strategy for plug-in hybrid electric vehicles
conference, December 2011

Approximate Dynamic Programming: Solving the Curses of Dimensionality
book, January 2007

Energy Management Strategy for a Hybrid Electric Vehicle Based on Deep Reinforcement Learning
journal, January 2018