Learning-based traffic signal control algorithms with neighborhood information sharing: An application for sustainable mobility
Abstract
Here, this research applies R-Markov Average Reward Technique based reinforcement learning (RL) algorithm, namely RMART, for vehicular signal control problem leveraging information sharing among signal controllers in connected vehicle environment. We implemented the algorithm in a network of 18 signalized intersections and compare the performance of RMART with fixed, adaptive, and variants of the RL schemes. Results show significant improvement in system performance for RMART algorithm with information sharing over both traditional fixed signal timing plans and real time adaptive control schemes. Additionally, the comparison with reinforcement learning algorithms including Q learning and SARSA indicate that RMART performs better at higher congestion levels. Further, a multi-reward structure is proposed that dynamically adjusts the reward function with varying congestion states at the intersection. Finally, the results from test networks show significant reduction in emissions (CO, CO2, NOx, VOC, PM10) when RL algorithms are implemented compared to fixed signal timings and adaptive schemes.
- Authors:
-
[1];
- Oak Ridge National Laboratory, Urban Dynamics Institute, 1 Bethel Valley Road, TN, USA
- Lyles School of Civil Engineering, Purdue University, 550 Stadium Mall Drive, West Lafayette, IN, USA
- Publication Date:
- Research Org.:
- Oak Ridge National Laboratory (ORNL), Oak Ridge, TN (United States)
- Sponsoring Org.:
- USDOE
- OSTI Identifier:
- 1415195
- Grant/Contract Number:
- AC05-00OR22725
- Resource Type:
- Accepted Manuscript
- Journal Name:
- Journal of Intelligent Transportation Systems
- Additional Journal Information:
- Journal Volume: 22; Journal Issue: 1; Journal ID: ISSN 1547-2450
- Publisher:
- Taylor & Francis
- Country of Publication:
- United States
- Language:
- English
- Subject:
- 97 MATHEMATICS AND COMPUTING; 33 ADVANCED PROPULSION SYSTEMS; 54 ENVIRONMENTAL SCIENCES; connected and automated vehicles; reinforcement learning; sustainable transportation; traffic signal control; vehicular emissions
Citation Formats
Aziz, H. M. Abdul, Zhu, Feng, and Ukkusuri, Satish V. Learning-based traffic signal control algorithms with neighborhood information sharing: An application for sustainable mobility. United States: N. p., 2017.
Web. doi:10.1080/15472450.2017.1387546.
Aziz, H. M. Abdul, Zhu, Feng, & Ukkusuri, Satish V. Learning-based traffic signal control algorithms with neighborhood information sharing: An application for sustainable mobility. United States. https://doi.org/10.1080/15472450.2017.1387546
Aziz, H. M. Abdul, Zhu, Feng, and Ukkusuri, Satish V. Mon .
"Learning-based traffic signal control algorithms with neighborhood information sharing: An application for sustainable mobility". United States. https://doi.org/10.1080/15472450.2017.1387546. https://www.osti.gov/servlets/purl/1415195.
@article{osti_1415195,
title = {Learning-based traffic signal control algorithms with neighborhood information sharing: An application for sustainable mobility},
author = {Aziz, H. M. Abdul and Zhu, Feng and Ukkusuri, Satish V.},
abstractNote = {Here, this research applies R-Markov Average Reward Technique based reinforcement learning (RL) algorithm, namely RMART, for vehicular signal control problem leveraging information sharing among signal controllers in connected vehicle environment. We implemented the algorithm in a network of 18 signalized intersections and compare the performance of RMART with fixed, adaptive, and variants of the RL schemes. Results show significant improvement in system performance for RMART algorithm with information sharing over both traditional fixed signal timing plans and real time adaptive control schemes. Additionally, the comparison with reinforcement learning algorithms including Q learning and SARSA indicate that RMART performs better at higher congestion levels. Further, a multi-reward structure is proposed that dynamically adjusts the reward function with varying congestion states at the intersection. Finally, the results from test networks show significant reduction in emissions (CO, CO2, NOx, VOC, PM10) when RL algorithms are implemented compared to fixed signal timings and adaptive schemes.},
doi = {10.1080/15472450.2017.1387546},
journal = {Journal of Intelligent Transportation Systems},
number = 1,
volume = 22,
place = {United States},
year = {Mon Apr 03 00:00:00 EDT 2017},
month = {Mon Apr 03 00:00:00 EDT 2017}
}
Free Publicly Available Full Text
Publisher's Version of Record
Other availability
Search WorldCat to find libraries that may hold this journal
Cited by: 34 works
Citation information provided by
Web of Science
Web of Science
Save to My Library
You must Sign In or Create an Account in order to save documents to your library.
Works referenced in this record:
Multiagent Reinforcement Learning for Urban Traffic Control Using Coordination Graphs
book, January 2008
- Kuyer, Lior; Whiteson, Shimon; Bakker, Bram
- Machine Learning and Knowledge Discovery in Databases
A real-time traffic signal control system: architecture, algorithms, and analysis
journal, December 2001
- Mirchandani, Pitu; Head, Larry
- Transportation Research Part C: Emerging Technologies, Vol. 9, Issue 6
A survey of vehicular communications for traffic signal optimization
journal, April 2015
- Florin, Ryan; Olariu, Stephan
- Vehicular Communications, Vol. 2, Issue 2
A Distributed Approach for Coordination of Traffic Signal Agents
journal, January 2005
- Bazzan, Ana L. C.
- Autonomous Agents and Multi-Agent Systems, Vol. 10, Issue 1
Towards multi-agent reinforcement learning for integrated network of optimal traffic controllers (MARLIN-OTC)
journal, April 2010
- El-Tantawy, Samah; Abdulhai, Baher
- Transportation Letters, Vol. 2, Issue 2
Neural Networks for Real-Time Traffic Signal Control
journal, September 2006
- Srinivasan, D.; Choy, M. C.; Cheu, R. L.
- IEEE Transactions on Intelligent Transportation Systems, Vol. 7, Issue 3
Reinforcement learning: Introduction to theory and potential for transport applications
journal, December 2003
- Abdulhai, Baher; Kattan, Lina
- Canadian Journal of Civil Engineering, Vol. 30, Issue 6
Coordinated transit signal priority supporting transit progression under Connected Vehicle Technology
journal, June 2015
- Hu, Jia; Park, Byungkyu Brian; Lee, Young-Jae
- Transportation Research Part C: Emerging Technologies, Vol. 55
A real-time adaptive signal control in a connected vehicle environment
journal, June 2015
- Feng, Yiheng; Head, K. Larry; Khoshmagham, Shayan
- Transportation Research Part C: Emerging Technologies, Vol. 55
Simulation-Based Optimization: Parametric Optimization Techniques and Reinforcement Learning
book, November 2010
- Gosavi, Abhijit
- Operations Research/Computer Science Interfaces Series
A Novel Signal-Scheduling Algorithm With Quality-of-Service Provisioning for an Isolated Intersection
journal, September 2008
- Wunderlich, R.; Elhanany, I.
- IEEE Transactions on Intelligent Transportation Systems, Vol. 9, Issue 3
A multivariable regulator approach to traffic-responsive network-wide signal control
journal, February 2002
- Diakaki, Christina; Papageorgiou, Markos; Aboudolas, Kostas
- Control Engineering Practice, Vol. 10, Issue 2
Assessing the Mobility and Environmental Benefits of Reservation-Based Intelligent Intersections Using an Integrated Simulator
journal, September 2012
- Huang, Shan; Sadek, Adel W.; Zhao, Yunjie
- IEEE Transactions on Intelligent Transportation Systems, Vol. 13, Issue 3
Urban traffic signal control using reinforcement learning agents
journal, January 2010
- Balaji, P. G.; German, X.; Srinivasan, D.
- IET Intelligent Transport Systems, Vol. 4, Issue 3
Reinforcement learning in neurofuzzy traffic signal control
journal, June 2001
- Bingham, Ella
- European Journal of Operational Research, Vol. 131, Issue 2
Reinforcement Learning for True Adaptive Traffic Signal Control
journal, May 2003
- Abdulhai, Baher; Pringle, Rob; Karakoulas, Grigoris J.
- Journal of Transportation Engineering, Vol. 129, Issue 3
Model-based average reward reinforcement learning
journal, April 1998
- Tadepalli, Prasad; Ok, DoKyeong
- Artificial Intelligence, Vol. 100, Issue 1-2
Metro Traffic Regulation by Adaptive Optimal Control
journal, December 2011
- Lin, Wei-Song; Sheu, Jih-Wen
- IEEE Transactions on Intelligent Transportation Systems, Vol. 12, Issue 4
Integration of MOVES and dynamic traffic assignment models for fine-grained transportation and air quality analyses
conference, June 2011
- Lin, Jane; Chiu, Yi-Chang; Vallamsundar, Suriya
- 2011 IEEE Forum on Integrated and Sustainable Transportation Systems (FISTS)
Reinforcement learning-based multi-agent system for network traffic signal control
journal, January 2010
- Arel, I.; Liu, C.; Urbanik, T.
- IET Intelligent Transport Systems, Vol. 4, Issue 2
Genetic reinforcement learning for cooperative traffic signal control
conference, January 1994
- Mikami, S.; Kakazu, Y.
- Proceedings of the First IEEE Conference on Evolutionary Computation. IEEE World Congress on Computational Intelligence
Cooperative Adaptive Cruise Control: A Reinforcement Learning Approach
journal, December 2011
- Desjardins, C.; Chaib-draa, B.
- IEEE Transactions on Intelligent Transportation Systems, Vol. 12, Issue 4
Reinforcement Learning Agents for Traffic Signal Control in Oversaturated Networks
conference, April 2012
- Medina, Juan C.; Benekohal, Rahim F.
- First Congress of Transportation and Development Institute (TDI), Transportation and Development Institute Congress 2011
Traffic Signal Optimization with Greedy Randomized Tabu Search Algorithm
journal, August 2012
- Hu, Ta-Yin; Chen, Li-Wen
- Journal of Transportation Engineering, Vol. 138, Issue 8
Learning in groups of traffic signals
journal, June 2010
- Bazzan, Ana L. C.; de Oliveira, Denise; da Silva, Bruno C.
- Engineering Applications of Artificial Intelligence, Vol. 23, Issue 4
Traffic signal networks simulator using emotional algorithm with individuality
conference, January 2001
- Ishihara, H.; Fukuda, T.
- ITSC 2001. 2001 IEEE Intelligent Transportation Systems. Proceedings (Cat. No.01TH8585)
Development and Evaluation of a Cooperative Vehicle Intersection Control Algorithm Under the Connected Vehicles Environment
journal, March 2012
- Lee, Joyoung; Park, Byungkyu
- IEEE Transactions on Intelligent Transportation Systems, Vol. 13, Issue 1
An Approach to Tune Fuzzy Controllers Based on Reinforcement Learning for Autonomous Vehicle Control
journal, September 2005
- Dai, X.; Li, C. K.; Rad, A. B.
- IEEE Transactions on Intelligent Transportation Systems, Vol. 6, Issue 3
The real-time urban traffic control system CRONOS: Algorithm and experiments
journal, February 2006
- Boillot, Florence; Midenet, Sophie; Pierrelée, Jean-Claude
- Transportation Research Part C: Emerging Technologies, Vol. 14, Issue 1
Dynamic Traffic Control with Fairness and Throughput Optimization Using Vehicular Communications
journal, September 2013
- Lien-Wu Chen, ; Sharma, P.
- IEEE Journal on Selected Areas in Communications, Vol. 31, Issue 9
A Multiple-Goal Reinforcement Learning Method for Complex Vehicle Overtaking Maneuvers
journal, June 2011
- Ngai, Daniel Chi Kit; Yung, Nelson Hon Ching
- IEEE Transactions on Intelligent Transportation Systems, Vol. 12, Issue 2
Reinforcement Learning With Function Approximation for Traffic Signal Control
journal, June 2011
- La, Prashanth; Bhatnagar, Shalabh
- IEEE Transactions on Intelligent Transportation Systems, Vol. 12, Issue 2
Q-learning
journal, May 1992
- Watkins, Christopher J. C. H.; Dayan, Peter
- Machine Learning, Vol. 8, Issue 3-4
An integrated modeling approach for facilitating emission estimations of alternative fueled vehicles
journal, January 2012
- Xie, Yuanchang; Chowdhury, Mashrur; Bhavsar, Parth
- Transportation Research Part D: Transport and Environment, Vol. 17, Issue 1
Evaluating the impacts of urban corridor traffic signal optimization on vehicle emissions and fuel consumption
journal, March 2012
- Kwak, Jaeyoung; Park, Byungkyu; Lee, Jaesup
- Transportation Planning and Technology, Vol. 35, Issue 2
Impact of Connected Vehicle on Work Zone Network Safety through Dynamic Route Guidance
journal, March 2016
- Genders, Wade; Razavi, Saiedeh N.
- Journal of Computing in Civil Engineering, Vol. 30, Issue 2
On Average Versus Discounted Reward Temporal-Difference Learning
journal, January 2002
- Tsitsiklis, John N.; Van Roy, Benjamin
- Machine Learning, Vol. 49, Issue 2/3, p. 179-191
Reinforcement Learning
book, July 2012
- Kulkarni, Parag
- Reinforcement and Systemic Machine Learning for Decision Making
A Multivariable Regulator Approach to Traffic-Responsive Network-Wide Signal Control
journal, June 2000
- Diakaki, Christina; Papageorgiou, Markos; Aboudolas, Kostas
- IFAC Proceedings Volumes, Vol. 33, Issue 9
A Distributed Approach for Coordination of Traffic Signal Agents
journal, March 2005
- Bazzan, Ana L. C.
- Autonomous Agents and Multi-Agent Systems, Vol. 10, Issue 2
Works referencing / citing this record:
Asynchronous n -step Q-learning adaptive traffic signal control
journal, January 2019
- Genders, Wade; Razavi, Saiedeh
- Journal of Intelligent Transportation Systems, Vol. 23, Issue 4
Optimizing multi-agent based urban traffic signal control system
journal, October 2018
- Xu, Mingtao; An, Kun; Vu, Le Hai
- Journal of Intelligent Transportation Systems, Vol. 23, Issue 4
Network-wide traffic signal control based on the discovery of critical nodes and deep reinforcement learning
journal, January 2019
- Xu, Ming; Wu, Jianping; Huang, Ling
- Journal of Intelligent Transportation Systems, Vol. 24, Issue 1
Cooperative Bargain for the Autonomous Separation of Traffic Flows in Smart Reversible Lanes
journal, October 2019
- Xiao, Guangbing; Zhang, Haibo; Sun, Ning
- Complexity, Vol. 2019