Learning-based traffic signal control algorithms with neighborhood information sharing: An application for sustainable mobility

Aziz, H. M.  Abdul; Zhu, Feng; Ukkusuri, Satish V.

doi:10.1080/15472450.2017.1387546

Title: Learning-based traffic signal control algorithms with neighborhood information sharing: An application for sustainable mobility

Journal Article · Mon Apr 03 00:00:00 EDT 2017 · Journal of Intelligent Transportation Systems

DOI:https://doi.org/10.1080/15472450.2017.1387546· OSTI ID:1415195

^[1]; Zhu, Feng ^[2]; Ukkusuri, Satish V. ^[2]

Oak Ridge National Laboratory, Urban Dynamics Institute, 1 Bethel Valley Road, TN, USA
Lyles School of Civil Engineering, Purdue University, 550 Stadium Mall Drive, West Lafayette, IN, USA

Here, this research applies R-Markov Average Reward Technique based reinforcement learning (RL) algorithm, namely RMART, for vehicular signal control problem leveraging information sharing among signal controllers in connected vehicle environment. We implemented the algorithm in a network of 18 signalized intersections and compare the performance of RMART with fixed, adaptive, and variants of the RL schemes. Results show significant improvement in system performance for RMART algorithm with information sharing over both traditional fixed signal timing plans and real time adaptive control schemes. Additionally, the comparison with reinforcement learning algorithms including Q learning and SARSA indicate that RMART performs better at higher congestion levels. Further, a multi-reward structure is proposed that dynamically adjusts the reward function with varying congestion states at the intersection. Finally, the results from test networks show significant reduction in emissions (CO, CO₂, NO_x, VOC, PM₁₀) when RL algorithms are implemented compared to fixed signal timings and adaptive schemes.

View Accepted Manuscript (DOE)

Cite

Export

Save

Research Organization:: Oak Ridge National Laboratory (ORNL), Oak Ridge, TN (United States)

Sponsoring Organization:: USDOE

Grant/Contract Number:: AC05-00OR22725

OSTI ID:: 1415195

Journal Information:: Journal of Intelligent Transportation Systems, Vol. 22, Issue 1; ISSN 1547-2450

Publisher:: Taylor & FrancisCopyright Statement

Country of Publication:: United States

Language:: English

Citation Metrics:

Cited by: 34 works

Citation information provided by
Web of Science

References (44)

Multiagent Reinforcement Learning for Urban Traffic Control Using Coordination Graphs Kuyer, Lior; Whiteson, Shimon; Bakker, Bram Machine Learning and Knowledge Discovery in Databases https://doi.org/10.1007/978-3-540-87479-9_61	book	January 2008
A real-time traffic signal control system: architecture, algorithms, and analysis Mirchandani, Pitu; Head, Larry Transportation Research Part C: Emerging Technologies, Vol. 9, Issue 6 https://doi.org/10.1016/S0968-090X(00)00047-4	journal	December 2001
A survey of vehicular communications for traffic signal optimization Florin, Ryan; Olariu, Stephan Vehicular Communications, Vol. 2, Issue 2 https://doi.org/10.1016/j.vehcom.2015.03.002	journal	April 2015
A Distributed Approach for Coordination of Traffic Signal Agents Bazzan, Ana L. C. Autonomous Agents and Multi-Agent Systems, Vol. 10, Issue 1 https://doi.org/10.1007/s10458-004-6975-9	journal	January 2005
Towards multi-agent reinforcement learning for integrated network of optimal traffic controllers (MARLIN-OTC) El-Tantawy, Samah; Abdulhai, Baher Transportation Letters, Vol. 2, Issue 2 https://doi.org/10.3328/TL.2010.02.02.89-110	journal	April 2010
Neural Networks for Real-Time Traffic Signal Control Srinivasan, D.; Choy, M. C.; Cheu, R. L. IEEE Transactions on Intelligent Transportation Systems, Vol. 7, Issue 3 https://doi.org/10.1109/TITS.2006.874716	journal	September 2006
Reinforcement learning: Introduction to theory and potential for transport applications Abdulhai, Baher; Kattan, Lina Canadian Journal of Civil Engineering, Vol. 30, Issue 6 https://doi.org/10.1139/l03-014	journal	December 2003
Coordinated transit signal priority supporting transit progression under Connected Vehicle Technology Hu, Jia; Park, Byungkyu Brian; Lee, Young-Jae Transportation Research Part C: Emerging Technologies, Vol. 55 https://doi.org/10.1016/j.trc.2014.12.005	journal	June 2015
A real-time adaptive signal control in a connected vehicle environment Feng, Yiheng; Head, K. Larry; Khoshmagham, Shayan Transportation Research Part C: Emerging Technologies, Vol. 55 https://doi.org/10.1016/j.trc.2015.01.007	journal	June 2015
Simulation-Based Optimization: Parametric Optimization Techniques and Reinforcement Learning Gosavi, Abhijit Operations Research/Computer Science Interfaces Series https://doi.org/10.1007/978-1-4757-3766-0	book	November 2010
A Novel Signal-Scheduling Algorithm With Quality-of-Service Provisioning for an Isolated Intersection Wunderlich, R.; Elhanany, I. IEEE Transactions on Intelligent Transportation Systems, Vol. 9, Issue 3 https://doi.org/10.1109/TITS.2008.928266	journal	September 2008
A multivariable regulator approach to traffic-responsive network-wide signal control Diakaki, Christina; Papageorgiou, Markos; Aboudolas, Kostas Control Engineering Practice, Vol. 10, Issue 2 https://doi.org/10.1016/S0967-0661(01)00121-6	journal	February 2002
Assessing the Mobility and Environmental Benefits of Reservation-Based Intelligent Intersections Using an Integrated Simulator Huang, Shan; Sadek, Adel W.; Zhao, Yunjie IEEE Transactions on Intelligent Transportation Systems, Vol. 13, Issue 3 https://doi.org/10.1109/TITS.2012.2186442	journal	September 2012
Urban traffic signal control using reinforcement learning agents Balaji, P. G.; German, X.; Srinivasan, D. IET Intelligent Transport Systems, Vol. 4, Issue 3 https://doi.org/10.1049/iet-its.2009.0096	journal	January 2010
Reinforcement learning in neurofuzzy traffic signal control Bingham, Ella European Journal of Operational Research, Vol. 131, Issue 2 https://doi.org/10.1016/S0377-2217(00)00123-5	journal	June 2001
Reinforcement Learning for True Adaptive Traffic Signal Control Abdulhai, Baher; Pringle, Rob; Karakoulas, Grigoris J. Journal of Transportation Engineering, Vol. 129, Issue 3 https://doi.org/10.1061/(ASCE)0733-947X(2003)129:3(278)	journal	May 2003
Model-based average reward reinforcement learning Tadepalli, Prasad; Ok, DoKyeong Artificial Intelligence, Vol. 100, Issue 1-2 https://doi.org/10.1016/S0004-3702(98)00002-2	journal	April 1998
Metro Traffic Regulation by Adaptive Optimal Control Lin, Wei-Song; Sheu, Jih-Wen IEEE Transactions on Intelligent Transportation Systems, Vol. 12, Issue 4 https://doi.org/10.1109/TITS.2011.2142306	journal	December 2011
Integration of MOVES and dynamic traffic assignment models for fine-grained transportation and air quality analyses Lin, Jane; Chiu, Yi-Chang; Vallamsundar, Suriya 2011 IEEE Forum on Integrated and Sustainable Transportation Systems (FISTS) https://doi.org/10.1109/FISTS.2011.5973657	conference	June 2011
Reinforcement learning-based multi-agent system for network traffic signal control Arel, I.; Liu, C.; Urbanik, T. IET Intelligent Transport Systems, Vol. 4, Issue 2 https://doi.org/10.1049/iet-its.2009.0070	journal	January 2010
Genetic reinforcement learning for cooperative traffic signal control Mikami, S.; Kakazu, Y. Proceedings of the First IEEE Conference on Evolutionary Computation. IEEE World Congress on Computational Intelligence https://doi.org/10.1109/ICEC.1994.350012	conference	January 1994
Cooperative Adaptive Cruise Control: A Reinforcement Learning Approach Desjardins, C.; Chaib-draa, B. IEEE Transactions on Intelligent Transportation Systems, Vol. 12, Issue 4 https://doi.org/10.1109/TITS.2011.2157145	journal	December 2011
Reinforcement Learning Agents for Traffic Signal Control in Oversaturated Networks Medina, Juan C.; Benekohal, Rahim F. First Congress of Transportation and Development Institute (TDI), Transportation and Development Institute Congress 2011 https://doi.org/10.1061/41167(398)14	conference	April 2012
Traffic Signal Optimization with Greedy Randomized Tabu Search Algorithm Hu, Ta-Yin; Chen, Li-Wen Journal of Transportation Engineering, Vol. 138, Issue 8 https://doi.org/10.1061/(ASCE)TE.1943-5436.0000404	journal	August 2012
Learning in groups of traffic signals Bazzan, Ana L. C.; de Oliveira, Denise; da Silva, Bruno C. Engineering Applications of Artificial Intelligence, Vol. 23, Issue 4 https://doi.org/10.1016/j.engappai.2009.11.009	journal	June 2010
Traffic signal networks simulator using emotional algorithm with individuality Ishihara, H.; Fukuda, T. ITSC 2001. 2001 IEEE Intelligent Transportation Systems. Proceedings (Cat. No.01TH8585) https://doi.org/10.1109/ITSC.2001.948804	conference	January 2001
Development and Evaluation of a Cooperative Vehicle Intersection Control Algorithm Under the Connected Vehicles Environment Lee, Joyoung; Park, Byungkyu IEEE Transactions on Intelligent Transportation Systems, Vol. 13, Issue 1 https://doi.org/10.1109/TITS.2011.2178836	journal	March 2012
An Approach to Tune Fuzzy Controllers Based on Reinforcement Learning for Autonomous Vehicle Control Dai, X.; Li, C. K.; Rad, A. B. IEEE Transactions on Intelligent Transportation Systems, Vol. 6, Issue 3 https://doi.org/10.1109/TITS.2005.853698	journal	September 2005
The real-time urban traffic control system CRONOS: Algorithm and experiments Boillot, Florence; Midenet, Sophie; Pierrelée, Jean-Claude Transportation Research Part C: Emerging Technologies, Vol. 14, Issue 1 https://doi.org/10.1016/j.trc.2006.05.001	journal	February 2006
Dynamic Traffic Control with Fairness and Throughput Optimization Using Vehicular Communications IEEE Journal on Selected Areas in Communications, Vol. 31, Issue 9 https://doi.org/10.1109/JSAC.2013.SUP.0513045	journal	September 2013
A Multiple-Goal Reinforcement Learning Method for Complex Vehicle Overtaking Maneuvers Ngai, Daniel Chi Kit; Yung, Nelson Hon Ching IEEE Transactions on Intelligent Transportation Systems, Vol. 12, Issue 2 https://doi.org/10.1109/TITS.2011.2106158	journal	June 2011
Reinforcement Learning With Function Approximation for Traffic Signal Control La, Prashanth; Bhatnagar, Shalabh IEEE Transactions on Intelligent Transportation Systems, Vol. 12, Issue 2 https://doi.org/10.1109/TITS.2010.2091408	journal	June 2011
Q-learning Watkins, Christopher J. C. H.; Dayan, Peter Machine Learning, Vol. 8, Issue 3-4 https://doi.org/10.1007/BF00992698	journal	May 1992
An integrated modeling approach for facilitating emission estimations of alternative fueled vehicles Xie, Yuanchang; Chowdhury, Mashrur; Bhavsar, Parth Transportation Research Part D: Transport and Environment, Vol. 17, Issue 1 https://doi.org/10.1016/j.trd.2011.08.009	journal	January 2012
Reinforcement Learning Sutton, Richard S. https://doi.org/10.1007/978-1-4615-3618-5	book	January 1992
Evaluating the impacts of urban corridor traffic signal optimization on vehicle emissions and fuel consumption Kwak, Jaeyoung; Park, Byungkyu; Lee, Jaesup Transportation Planning and Technology, Vol. 35, Issue 2 https://doi.org/10.1080/03081060.2011.651877	journal	March 2012
Impact of Connected Vehicle on Work Zone Network Safety through Dynamic Route Guidance Genders, Wade; Razavi, Saiedeh N. Journal of Computing in Civil Engineering, Vol. 30, Issue 2 https://doi.org/10.1061/(ASCE)CP.1943-5487.0000490	journal	March 2016
On Average Versus Discounted Reward Temporal-Difference Learning Tsitsiklis, John N.; Van Roy, Benjamin Machine Learning, Vol. 49, Issue 2/3, p. 179-191 https://doi.org/10.1023/A:1017980312899	journal	January 2002
Reinforcement Learning Kubat, Miroslav An Introduction to Machine Learning https://doi.org/10.1007/978-3-319-63913-0_17	book	January 2017
Reinforcement Learning Kulkarni, Parag Reinforcement and Systemic Machine Learning for Decision Making https://doi.org/10.1002/9781118266502.ch3	book	July 2012
A Multivariable Regulator Approach to Traffic-Responsive Network-Wide Signal Control Diakaki, Christina; Papageorgiou, Markos; Aboudolas, Kostas IFAC Proceedings Volumes, Vol. 33, Issue 9 https://doi.org/10.1016/s1474-6670(17)38203-4	journal	June 2000
A Distributed Approach for Coordination of Traffic Signal Agents Bazzan, Ana L. C. Autonomous Agents and Multi-Agent Systems, Vol. 10, Issue 2 https://doi.org/10.1023/b:agnt.0000049887.90232.cd	journal	March 2005
Reinforcement Learning Isiadinso, Ebuka Unpublished https://doi.org/10.13140/rg.2.2.25833.11364	text	January 2018
Reinforcement Learning Weber, Cornelius; Elshaw, Mark; Mayer, N. Michael https://doi.org/10.5772/54	book	January 2008

Cited By (4)

Asynchronous n -step Q-learning adaptive traffic signal control Genders, Wade; Razavi, Saiedeh Journal of Intelligent Transportation Systems, Vol. 23, Issue 4 https://doi.org/10.1080/15472450.2018.1491003	journal	January 2019
Optimizing multi-agent based urban traffic signal control system Xu, Mingtao; An, Kun; Vu, Le Hai Journal of Intelligent Transportation Systems, Vol. 23, Issue 4 https://doi.org/10.1080/15472450.2018.1501273	journal	October 2018
Network-wide traffic signal control based on the discovery of critical nodes and deep reinforcement learning Xu, Ming; Wu, Jianping; Huang, Ling Journal of Intelligent Transportation Systems, Vol. 24, Issue 1 https://doi.org/10.1080/15472450.2018.1527694	journal	January 2019
Cooperative Bargain for the Autonomous Separation of Traffic Flows in Smart Reversible Lanes Xiao, Guangbing; Zhang, Haibo; Sun, Ning Complexity, Vol. 2019 https://doi.org/10.1155/2019/2893732	journal	October 2019

Similar Records

Investigating the Impact of Connected Vehicle Market Share on the Performance of Reinforcement-Learning Based Traffic Signal Control

Technical Report · Sun Jun 30 00:00:00 EDT 2019 · OSTI ID:1415195

Aziz, H M Abdul; Wang, Hong; Young, Stanley; +1 more

A junction-tree based learning algorithm to optimize network wide traffic control: A coordinated multi-agent framework

Journal Article · Sat Jan 31 00:00:00 EST 2015 · Transportation Research Part C: Emerging Technologies · OSTI ID:1415195

Zhu, Feng; Aziz, H. M. Abdul; Qian, Xinwu; +1 more

Minimizing Energy Consumption from Connected Signalized Intersections by Reinforcement Learning

Conference · Mon Dec 10 00:00:00 EST 2018 · OSTI ID:1415195

Bin Al Islam, S. M. A.; Aziz, H. M. Abdul; Wang, Hong; +1 more

Related Subjects

97 MATHEMATICS AND COMPUTING
33 ADVANCED PROPULSION SYSTEMS
54 ENVIRONMENTAL SCIENCES
connected and automated vehicles
reinforcement learning
sustainable transportation
traffic signal control
vehicular emissions

Title: Learning-based traffic signal control algorithms with neighborhood information sharing: An application for sustainable mobility

Citation Formats

References (44)

Cited By (4)

Similar Records

Related Subjects