کنترل انعطاف‌پذیر سیگنال ترافیک مبتنی بر توسعه نمایش حالت در روش یادگیری تقویتی عمیق در هنگام وقوع تصادف در تقاطعات شهری

نوع مقاله : علمی - پژوهشی

نویسندگان

1 دانشجوی دکترا، گروه مهندسی کنترل، دانشکده مهندسی برق و کامپیوتر، دانشگاه تربیت مدرس، ایران

2 دانشیار، گروه مهندسی کنترل، دانشکده مهندسی برق و کامپیوتر، دانشگاه تربیت مدرس، ایران

چکیده

روش‌های یادگیری تقویتی عمیق نتایج امیدوارکننده‌ای را در توسعه کنترل‌کننده‌های سیگنال ترافیک نشان داده‌اند. در این مقاله، انعطاف‌پذیری یک کنترل­ کننده مبتنی بر یادگیری تقویتی عمیق را در شرایط ترافیک با حجم زیاد و تحت طیف وسیعی از اختلالات محیطی مانند تصادفات، بررسی کرده و یک کنترل‌کننده قابل اعتماد را در محیط با ترافیک پویا پیشنهاد می ­دهیم. در این روش ،با استفاده از رویکرد گسسته ­سازی هر یک از خیابان­ های چهارراه به سلول­ هایی تقسیم شده وتاثیر اندازه این سلول­ ها به لحاظ متفاوت بودن یا یکسان بودن با یکدیگردر کارآیی الگوریتم بررسی می­ گردد. با انتخاب یک فضای حالت توسعه یافته و متراکم، اطلاعاتی به عامل به عنوان ورودی داده می­ شودکه بتواند درک کاملی از محیط را در اختیار عامل قرار دهد. برای آموزش عامل از روش یادگیری عمیق Q و بازپخش تجربه استفاده شده و مدل پیشنهادی در شبیه ­ساز ترافیک SUMO ارزیابی شده است. نتایج شبیه‌سازی کارایی روش پیشنهادی را در کاهش طول صف حتی در صورت وجود اختلال تأیید می‌کند.

کلیدواژه‌ها


عنوان مقاله [English]

Resilient Traffic Signal Control Based on the Development of State Definition in the Deep Reinforcement Learning Method in the Presence of Accident at Urban Intersections

نویسندگان [English]

  • Zahra Zeinaly 1
  • Mahdi Sojoodi 2
1 P.hD. student, Department of Control Engineering, School of Electrical and Computer Engineering, Tarbiat Modares University, Tehran, Iran
2 Associated Professor, Department of Control Engineering, School of Electrical and Computer Engineering, Tarbiat Modares University, Tehran, Iran
چکیده [English]

Deep reinforcement learning methods have shown promising results in the development of traffic signal controllers. In this paper, we evaluate the flexibility of a controller based on Deep Reinforcement Learning under high traffic volume and under a variety of environmental disruptions, such as accidents, and propose a reliable controller in a dynamic traffic environment. In this method, using the discretization approach, each of the intersection roads is divided into cells and the effect of the size of these cells in terms of whether they are different or identical is studied on the efficiency of the algorithm. By selecting an extended and dense state space, the agent is given information as input that can provide it with a complete understanding of the environment. The Q-deep learning method and experience replay are used to train the agent, and the proposed model is evaluated in the SUMO traffic simulator. The simulation results confirm the efficiency of the proposed method in reducing the queue length even in the presence of a disruption.

کلیدواژه‌ها [English]

  • Traffic safety
  • Accident
  • Traffic control
  • Deep reinforcement learning
- مهماندار، م.، آریانا، م.، مبادری، ت. و خلیلی، ا. (1399) “ ارزیابی مولفه های موثر بر ارتقای فرهنگ ایمنی ترافیک و کاهش تلفات با موتورسیکلت”، فصلنامه مهندسی حمل و نقل، سال یازدهم، شماره سوم، ص.649-663.
 
- Bálint, K., Tamás, T. and Tamás, B. (2022) “Deep Reinforcement Learning based approach for Traffic Signal Control”. Transportation Research Procedia. Vol. 62, pp. 278–285.
 
- Casas N. (2017) “Deep deterministic policy gradient for urban traffic light control”. arXiv preprint arXiv:170309035.
 
- Chu, K.F., Lam, A.Y. and Li, V.O. (2021) “Traffic Signal Control Using End-to-End Off-Policy Deep Reinforcement Learning”. IEEE Transactions on Intelligent Transportation Systems.
 
- Chu, T., Wang, J., Codecà, L. and Li, Z. (2019) “Multi-agent deep reinforcement learning for large-scale traffic signal control”. IEEE Transactions on Intelligent Transportation Systems. Vol. 21, No. 3, pp. 1086–1095.
 
- Essa, M. and Sayed, T. (2020) “Self-learning adaptive traffic signal control for real-time safety optimization”. Accident Analysis & Prevention. Vol. 146:105713.
 
- Gao, J., Shen, Y., Liu, J., Ito, M. and Shiratori, N. (2017) “Adaptive traffic signal control: Deep reinforcement learning algorithm with experience replay and target network”. arXiv preprint arXiv:170502755.
 
- Genders, W. and Razavi., S. (2016) “Using a deep reinforcement learning agent for traffic signal control”. arXiv preprint arXiv:161101142.
 
- Gong, Y., Abdel-Aty, M., Yuan, J. and Cai, Q. (2020) “Multi-objective reinforcement learning approach for improving safety at intersections with adaptive traffic signal control”. Accident Analysis & Prevention. Vol. 144:105655.
 
- Jamil. ARM., Ganguly. KK. and Nower. N (2021) “Adaptive traffic signal control system using composite reward architecture based deep reinforcement learning”. IET Intelligent Transport Systems.  Vol. 14, No. 14, pp. 2030–2041.
 
- Krajzewicz, D., Erdmann, J., Behrisch, M. and Bieker, L. (2012) “Recent development and applications of SUMO-Simulation of Urban MObility”. International journal on advances in systems and measurements. Vol. 5(3 & 4).
 
- Li, L., Lv, Y. and Wang, F.Y. (2016) “Traffic signal timing via deep reinforcement learning”. IEEE/CAA Journal of Automatica Sinica. Vol. 3, No.3, pp. 247–254.
 
- Li, M., Li, Z., Xu, C. and Liu, T. (2020) “Deep reinforcement learning-based vehicle driving strategy to reduce crash risks in traffic oscillations. Transportation research record”. Vol. 2674, No. 10, pp. 42–54.
 
- Liang, X., Du, X., Wang, G. and Han, Z. (2018) “Deep reinforcement learning for traffic light control in vehicular networks”. arXiv preprint arXiv:180311115.
 
- Liang, X., Du, X., Wang, G and Han, Z. (2019) “A deep q learning network for traffic lights’ cycle control in vehicular networks”. IEEE Transactions on Vehicular Technology. Vol. 68, No. 2, pp. 1243–1253.
 
- Maurya, A.K., Dey, S. and Das, S. (2015) “Speed and time headway distribution under mixed traffic condition”. Journal of the Eastern Asia Society for Transportation Studies. Vol, 11. pp. 1774–1792.
 
- Mousavi, S.S., Schukat M. and Howley, E. (2017)  “Traffic light control using deep policy-gradient and value-function-based reinforcement learning”. IET Intelligent Transport Systems. Vol. 11, No. 7, pp. 417–423.
 
- Paul, A. and Mitra, S. (2022) “Exploring reward efficacy in traffic management using deep reinforcement learning in intelligent transportation system”. ETRI Journal. Vol. 44, No. 2, pp. 194–207.
 
- Rodrigues, F. and Azevedo, C.L. (2019) “Towards robust deep reinforcement learning for traffic signal control: Demand surges, incidents and sensor failures”.  IEEE intelligent transportation systems conference (ITSC). Auckland, New Zealand: 27-30 October 2019.
 
- Sutton, R.S.and Barto, A.G. (2018) “Reinforcement learning: An introduction”. MIT press.
 
- Van der Pol, E., Oliehoek, F.A. (2016) “Coordinated deep reinforcement learners for traffic light control”. Proceedings of Learning, Inference and Control of Multi-Agent Systems (NIPS).
 
- Vidali, A., Crociani, L., Vizzari, G. and Bandini, S. (2019) “A Deep Reinforcement Learning Approach to Adaptive Traffic Lights Management”. In: WOA. Parma, Italy, 26-28 June 2019.
 
- Wang. T., Cao. J. and. Hussain A (2021) “Adaptive Traffic Signal Control for large-scale scenario with Cooperative Group-based Multi-agent reinforcement learning”. Transportation research part C: emerging technologies. Vol. 125:103046.
 
- Wei. H., Zheng, G., Gayah, V. and Li, Z. (2021) “Recent advances in reinforcement learning for traffic signal control: A survey of models and evaluation”. ACM SIGKDD Explorations Newsletter. Vol. 22, No. 2, pp. 12–18.
 
- Wei, H., Zheng, G., Yao, H. and Li, Z. (2018) “Intellilight: A reinforcement learning approach for intelligent traffic light control”. In: Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. pp. 2496–2505.
 
- Yoon, J., Ahn, K., Park, J and Yeo, H. (2021) “Transferable traffic signal control: Reinforcement learning with graph centric state representation”. Transportation Research Part C: Emerging Technologies. Vol. 130:103321.
 
- Roy, A., Hossain, M. and Muromachi, Y. (2022) “A deep reinforcement learning-based intelligent intervention framework for real-time proactive road safety management”. Accident Analysis & Prevention. Vol. 165, pp. 106512.
 
- Zheng, G., Zang, X., Xu, N., Wei, H., Yu, Z., Gayah, V., Xu, K. and Li Z. (2019) “Diagnosing reinforcement learning for traffic signal control”. arXiv preprint arXiv:190504716.