کنترل شتاب طولی و جانبی خودروی بدون سرنشین با استفاده از یادگیری تقویتی عمیق

نوع مقاله : علمی - پژوهشی

نویسندگان
1 دانشجوی دکتری، گروه مهندسی برق، دانشکده مهندسی برق و کامپیوتر، دانشگاه صنعتی مالک اشتر، تهران، ایران
2 استادیار، گروه مهندسی برق، دانشکده مهندسی برق، دانشگاه صنعتی مالک اشتر، تهران، ایران
3 استادیار، گروه مهندسی برق، دانشکده مهندسی برق و کامپیوتر، دانشگاه صنعتی مالک اشتر، تهران، ایران
چکیده
کنترل دقیق شتاب طولی و جانبی در خودروهای بدون سرنشین یکی از چالش‌های کلیدی در توسعه سیستم‌های خودران ایمن و کارآمد به شمار می‌رود. با توجه به رشد سریع فناوری‌های خودران و نیاز به بهبود عملکرد این سیستم‌ها در شرایط مختلف جاده‌ای، پژوهش در زمینه کنترل هوشمند این خودروها اهمیت ویژه‌ای یافته است. در این مقاله، یک روش نوآورانه برای کنترل شتاب طولی و جانبی خودروهای بدون سرنشین مبتنی بر یادگیری تقویتی ارائه شده است. روش پیشنهادی با بهره‌گیری از معماری تفکیک‌شده، پیچیدگی محاسباتی را کاهش داده و امکان استفاده از الگوریتم‌های یادگیری تقویتی پیشرفته‌تر را فراهم می‌کند. در مرحله اولیه، یک عامل یادگیری تقویتی واحد برای کنترل هم‌زمان شتاب طولی و جانبی طراحی و آموزش داده می‌شود. سپس، به‌منظور بهبود کارایی، عامل‌های یادگیری تقویتی به دو بخش مستقل برای کنترل طولی و جانبی تفکیک‌شده و به‌صورت جداگانه آموزش داده می‌شوند. نتایج شبیه‌سازی‌ها نشان می‌دهد که این تفکیک نه‌تنها سرعت همگرایی فرآیند آموزش را افزایش می‌دهد، بلکه دقت عملکرد سیستم کنترل را نیز به‌طور قابل توجهی بهبود می‌بخشد. جداسازی عامل­ها، کاهش حدود 2 ساعت از زمان آموزش شبکه­های عمیق،  تقلیل 38.2 درصدی میانگین خطای سمت و 10.1 درصدی میانگین خطای فاصله را نتیجه می­دهد. این یافته‌ها می‌تواند به توسعه سیستم‌های کنترلی پیشرفته‌تر و ایمن‌تر برای خودروهای خودران کمک کند و نقش مهمی در ارتقاء ایمنی و کارایی این فناوری‌ها ایفا نماید.
کلیدواژه‌ها
موضوعات

عنوان مقاله English

Longitudinal and Lateral Acceleration Control of Unmanned Vehicle Using Deep Reinforcement Learning

نویسندگان English

Mohsen Ebrahimi 1
FIROUZ ALLAHVERDIZADEH 2
Abdolreza Kashani Nia 3
1 Ph.D., Faculty of Electrical and Computer Engineering, Malek Ashtar University of Technology, Tehran, Iran
2 Assistant Professor, Faculty of Electrical and Computer Engineering, Malek Ashtar University of Technology, Tehran, Iran
3 Assistant Professor, Faculty of Electrical and Computer Engineering, Malek Ashtar University of Technology, Tehran, Iran
چکیده English

Accurate control of longitudinal and lateral acceleration in unmanned vehicles is one of the key challenges in the development of safe and efficient autonomous systems. Given the rapid growth of autonomous technologies and the need to improve the performance of these systems in various road conditions, research in the field of intelligent control of these vehicles has gained special importance. In this paper, an innovative method for controlling the longitudinal and lateral acceleration of unmanned vehicles based on reinforcement learning is presented. The proposed method, by utilizing a disaggregated architecture, reduces the computational complexity and enables the use of more advanced reinforcement learning algorithms. In the initial stage, a single reinforcement learning agent is designed and trained to simultaneously control longitudinal and lateral acceleration. Then, in order to improve efficiency, the reinforcement learning agents are separated into two independent parts for longitudinal and lateral control, and trained separately. Simulation results show that this separation not only increases the convergence speed of the training process, but also significantly improves the accuracy of the control system's performance. Separating the agents results in a reduction of about 2 hours of deep network training time, a 38.2% reduction in average direction error and a 10.1% reduction in average distance error. These findings could help develop more advanced and safer control systems for autonomous vehicles and play an important role in improving the safety and efficiency of these technologies.

کلیدواژه‌ها English

Unmanned vehicle
reinforcement learning
vehicle lateral acceleration
vehicle longitudinal acceleration
- Artuñedo, A., Moreno-Gonzalez, M., & Villagra, J. (2024). “Lateral control for autonomous vehicles: A comparative evaluation”. Annual reviews in control, 57, 100910.
 
- Baheri, A., Kolmanovsky, I., Girard, A., Tseng, H. E., & Filev, D. (2020, July) “Vision-based autonomous driving: A model learning approach”. In 2020 American Control Conference (ACC) (pp. 2520-2525). IEEE.
 
- Bay, O. (2021) “Abi research forecasts 8 million vehicles to ship with sae level 3, 4 and 5 autonomous technology in 2025”.
 
- Bellotti, F., Lazzaroni, L., Capello, A., Cossu, M., De Gloria, A., & Berta, R. (2023) “Explaining a deep reinforcement learning (DRL)-based automated driving agent in highway simulations”. IEEE Access, 11, 28522-28550.
 
- Cao, Y., Ni, K., Jiang, X., Kuroiwa, T., Zhang, H., Kawaguchi, T., & Jiang, W. (2023). Path following for autonomous ground vehicle using DDPG algorithm: a reinforcement learning approach. Applied Sciences, 13(11), 6847.
 
- Chen, J., Zhang, C., Luo, J., Xie, J., & Wan, Y. (2020). “Driving maneuvers prediction based autonomous driving control by deep Monte Carlo tree search”. IEEE transactions on vehicular technology, 69(7), 7146-7158.
 
- Claussmann, L., Revilloud, M., Gruyer, D., & Glaser, S. (2019) “A review of motion planning for highway autonomous driving”. IEEE Transactions on Intelligent Transportation Systems, 21(5), 1826-1848.
 
- Du, Y., Chen, J., Zhao, C., Liao, F., & Zhu, M. (2023). “A hierarchical framework for improving ride comfort of autonomous vehicles via deep reinforcement learning with external knowledge”. Computer‐Aided Civil and Infrastructure Engineering, 38(8), 1059-1078.
 
- Ebrahimi, M. (2023). “Broken rail detection with texture image processing using two-dimensional gray level co-occurrence matrix”. arXiv preprint arXiv:2304.11592.
 
- Ebrahimi, M., & Asgari, M. (2021). “Robust fractional-order fixed-structure controller design for uncertain non-commensurate fractional plants using fractional Kharitonov theorem”. Proceedings of the Institution of Mechanical Engineers, Part I: Journal of Systems and Control Engineering, 235(8), 1375-1387.
- Ebrahimi, M., & Nasrollahi, S. (2025). “Fractional Guidance Law with Impact Angle Constraint and Seeker’s Look Angle Limits”. Unmanned Systems, 13(01), 261-277.
- El Hamdani, S., Benamar, N., & Younis, M. (2020) “Pedestrian support in intelligent transportation systems: challenges, solutions and open issues”. Transportation research part C: emerging technologies, 121, 102856.
 
- Feng, D., Haase-Schütz, C., Rosenbaum, L., Hertlein, H., Glaeser, C., Timm, F., ... & Dietmayer, K. (2020) “Deep multi-modal object detection and semantic segmentation for autonomous driving: Datasets, methods, and challenges”. IEEE Transactions on Intelligent Transportation Systems, 22(3), 1341-1360.
 
- Grigorescu, S., Trasnea, B., Cocias, T., & Macesanu, G. (2020) “A survey of deep learning techniques for autonomous driving”. Journal of field robotics, 37(3), 362-386.
 
- Guo, J., Cheng, S., & Liu, Y. (2020) “Merging and diverging impact on mixed traffic of regular and autonomous vehicles”. IEEE Transactions on Intelligent Transportation Systems, 22(3), 1639-1649.
 
- Hoveidafard, A., Fardmoradinia, S., Golchin, B. and Ghaffari, A. (2023) "Simulation of autonomous vehicles using machine learning methods". International Journal of Transportation Engineering, 15(2), 3483-3507 (in  persian).
 
- Huang, Z., Zhang, J., Tian, R., & Zhang, Y. (2019) “End-to-end autonomous driving decision based on deep reinforcement learning”. In 2019 5th International Conference on Control, Automation and Robotics (ICCAR) (pp. 658-662). IEEE.
 
  -Kuutti, S., Bowden, R., Jin, Y., Barber, P., & Fallah, S. (2020) “A survey of deep learning applications to autonomous vehicle control”. IEEE Transactions on Intelligent Transportation Systems, 22(2), 712-733.
 
- Lamssaggad, A., Benamar, N., Hafid, A. S., & Msahli, M. (2021) “A survey on the current security landscape of intelligent transportation systems”. IEEE Access, 9, 9180-9208.
 
-  Lee, K., Isele, D., Theodorou, E. A., & Bae, S. (2022) “Spatiotemporal costmap inference for MPC via deep inverse reinforcement learning”. IEEE Robotics and Automation Letters, 7(2), 3194-3201.
 
- Li, Z., Yuan, S., Yin, X., Li, X., & Tang, S. (2023) “Research into autonomous vehicles following and obstacle avoidance based on deep reinforcement learning method under map constraints”. Sensors, 23(2), 844.
 
-Liu, Y., & Diao, S. (2024) “An automatic driving trajectory planning approach in complex traffic scenarios based on integrated driver style inference and deep reinforcement learning”. PLoS one, 19(1), e0297192.
 
 - Ma, Y., Wang, Z., Yang, H., & Yang, L. (2020) “Artificial intelligence applications in the development of autonomous vehicles: A survey”. IEEE/CAA Journal of Automatica Sinica, 7(2), 315-329.
 
- Mahaadevan, V. C., Narayanamoorthi, R., Panda, S., Dutta, S., & Dooly, G. (2024) “AViTRoN: Advanced vision track routing and navigation for autonomous charging of electric vehicles”. IEEE Access.
 
- Mao, Z., Liu, Y., & Qu, X. (2024) “Integrating big data analytics in autonomous driving: An unsupervised hierarchical reinforcement learning approach”. Transportation Research Part C: Emerging Technologies, 162, 104606.
 
- Ning, H., Yin, R., Ullah, A., & Shi, F. (2021) “A survey on hybrid human-artificial intelligence for autonomous driving”. IEEE Transactions on Intelligent Transportation Systems, 23(7), 6011-6026.
 
- Peličić, D., Ristić, B., & Radević, S. (2024) “Factors for the occurrence of road traffic injuries and better care of injured persons”. Zdravstvena zaštita, 53, 59.
 
- Pérez-Gil, Ó., Barea, R., López-Guillén, E., Bergasa, L. M., Gómez-Huélamo, C., Gutiérrez, R., & Díaz-Díaz, A. (2022) “Deep reinforcement learning based control for Autonomous Vehicles in CARLA”. Multimedia Tools and Applications, 81(3), 3553-3576.
 
- Rajamani, R. (2006). Vehicle dynamics and control. Boston, MA: Springer US.
 
- Singh, S. (2015) “Critical reasons for crashes investigated in the national motor vehicle crash causation survey” (No. DOT HS 812 115).
 
- Toromanoff, M., Wirbel, E., & Moutarde, F. (2020) “End-to-end model-free reinforcement learning for urban driving using implicit affordances”. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 7153-7162).
 
 
- Van Hasselt, H., Guez, A., & Silver, D. (2016) “Deep reinforcement learning with double q-learning”. In Proceedings of the AAAI conference on artificial intelligence (Vol. 30, No. 1).
 
 - Wu, Y., Tan, H., Qin, L., & Ran, B. (2020) “Differential variable speed limits control for freeway recurrent bottlenecks via deep actor-critic algorithm”. Transportation research part C: emerging technologies, 117, 102649.
 
- Wang, C., Cui, X., Zhao, S., Zhou, X., Song, Y., Wang, Y., & Guo, K. (2024) “A deep reinforcement learning-based active suspension control algorithm considering deterministic experience tracing for autonomous vehicle”. Applied Soft Computing, 153, 111259.
 
- Ye, F., Cheng, X., Wang, P., Chan, C. Y., & Zhang, J. (2020) “Automated lane change strategy using proximal policy optimization-based deep reinforcement learning”. In 2020 IEEE Intelligent Vehicles Symposium (IV) (pp. 1746-1752). IEEE.
 
- Yuan, M., Shan, J., & Mi, K. (2023) “From Naturalistic Traffic Data to Learning-Based Driving Policy: A Sim-to-Real Study”. IEEE Transactions on Vehicular Technology.
 
- Yusuf, S. A., Khan, A., & Souissi, R. (2024) “Vehicle-to-everything (V2X) in the autonomous vehicles domain–A technical review of communication, sensor, and AI technologies for road user safety”. Transportation Research Interdisciplinary Perspectives, 23, 100980.
 
- Zhang, Y., Sun, P., Yin, Y., Lin, L., & Wang, X. (2018) “Human-like autonomous vehicle speed control by deep reinforcement learning with double Q-learning”. In 2018 IEEE intelligent vehicles symposium (IV) (pp. 1251-1256). IEEE.
 
- Zhu, M., Wang, Y., Pu, Z., Hu, J., Wang, X., & Ke, R. (2020) “Safe, efficient, and comfortable velocity control based on reinforcement learning for autonomous driving”. Transportation Research Part C: Emerging Technologies, 117, 102662.
دوره 17، شماره 2 - شماره پیاپی 67
زمستان 1404
صفحه 5333-5358

  • تاریخ دریافت 21 بهمن 1403
  • تاریخ بازنگری 11 مرداد 1404
  • تاریخ پذیرش 16 شهریور 1404