Recurrent Proximal Policy Optimization Based Tractor-Trailer Wheeled Robot Automatic Parking Algorithm
DOI:
https://doi.org/10.14738/tecs.114.15355Keywords:
Tractor–trailer Wheeled Robot (TTWR), Proximal Policy Optimization, Recurrent Neural Network, Trajectory Planning, Obstacle AvoidanceAbstract
Truck-trailer reverse parking poses significant challenges due to the system's inherent instability, complex road geometry, and collision avoidance requirement. Traditional approaches for trailer control rely on manually designed control policies, which may have limited applicability and scalability. This paper presents a novel approach for autonomous reverse parking of a tractor-trailer wheeled robot (TTWR) system in tight and complex environments. Utilizing the Proximal Policy Optimization (PPO) algorithm, the controller is trained, with a long short-term memory (LSTM) network enhancing the handling of sequential observation data. To improve safety and reliability, ultrasonic sensors are installed on the trailer to detect nearby obstacles and ensure safety. Furthermore, a novel reward function is introduced that encourages the TTWR to maintain a safe distance from surrounding obstacles while minimizing the parking trajectory distance and the steering maneuver. The effectiveness of the proposed LSTM-PPO algorithm is compared with the original PPO and other trending reinforcement learning algorithms. The results demonstrate the improved convergence stability and speed of the proposed approach, as well as its successful execution of end-to-end reverse parking maneuvers in unknown environments.
References
J. Chen, B. Yuan, and M. Tomizuka, “Model-free Deep Reinforcement Learning for Urban Autonomous Driving.” arXiv, Oct. 21, 2019. Accessed: Jun. 03, 2023. [Online]. Available: http://arxiv.org/abs/1904.09503
A. Shaout, A. Shaout and S. A. Varagiri, "ADAS Technology Classifications Using a Modified Agile Process," 2022 International Arab Conference on Information Technology (ACIT), Abu Dhabi, United Arab Emirates, 2022, pp. 1-7, doi: 10.1109/ACIT57182.2022.9994175.
Q. Liu, X. Li, Y. Zhu, and S. Li, “A Review of Tracking Control Method for Tractor-Trailer Vehicles,” in Proceedings of the 2022 2nd International Conference on Control and Intelligent Robotics, Nanjing China: ACM, Jun. 2022, pp. 863–867. DOI: 10.1145/3548608.3559323.
L. Xu, E. Tseng, T. Pilutti, and S. Schondorf, “Yaw Rate Based Trailer Hitch Angle Estimation for Trailer Backup Assist,” presented at the WCX™ 17: SAE World Congress Experience, Mar. 2017, pp. 2017-01–0027. doi: 10.4271/2017-01-0027.
J. Jing, J. M. Maroli, Y. B. Salamah, M. Hejase, L. Fiorentini, and U. Ozguner, “Control Method Designs and Comparisons for Tractor-Trailer Vehicle Backward Path Tracking,” in 2019 American Control Conference (ACC), Philadelphia, PA, USA: IEEE, Jul. 2019, pp. 5531–5537. doi: 10.23919/ACC.2019.8815188.
F. Ye, X. Cheng, P. Wang, C.-Y. Chan, and J. Zhang, “Automated Lane Change Strategy using Proximal Policy Optimization-based Deep Reinforcement Learning,” in 2020 IEEE Intelligent Vehicles Symposium (IV), Las Vegas, NV, USA: IEEE, Oct. 2020, pp. 1746–1752. doi: 10.1109/IV47402.2020.9304668.
E. Bejar and A. Moran, “A Preview Neuro-Fuzzy Controller Based on Deep Reinforcement Learning for Backing Up a Truck-Trailer Vehicle,” in 2019 IEEE Canadian Conference of Electrical and Computer Engineering (CCECE), Edmonton, AB, Canada: IEEE, May 2019, pp. 1–4. doi: 10.1109/CCECE.2019.8861534.
S. L. Brunton and J. N. Kutz, Data-Driven Science and Engineering: Machine Learning, Dynamical Systems, and Control. Cambridge, UK etc.: Cambridge University Press, 2022.
C. Chen, Y. Liu, S. Kreiss, and A. Alahi, “Crowd-Robot Interaction: Crowd-Aware Robot Navigation With Attention-Based Deep Reinforcement Learning,” in 2019 International Conference on Robotics and Automation (ICRA), Montreal, QC, Canada: IEEE, May 2019, pp. 6015–6022. DOI: 10.1109/ICRA.2019.8794134.
Z. Du, Q. Miao, and C. Zong, “Trajectory Planning for Automated Parking Systems Using Deep Reinforcement Learning,” Int.J Automot. Technol., vol. 21, no. 4, pp. 881–887, Aug. 2020, doi: 10.1007/s12239-020-0085-9.
E. Bejar and A. Moran, “Reverse Parking a Car-Like Mobile Robot with Deep Reinforcement Learning and Preview Control,” in 2019 IEEE 9th Annual Computing and Communication Workshop and Conference (CCWC), Las Vegas, NV, USA: IEEE, Jan. 2019, pp. 0377–0383. DOI: 10.1109/CCWC.2019.8666613.
E. E. Reber, R. L. Michell, and C. J. Carter, ``Oxygen absorption in the earth's atmosphere,'' Aerospace Corp., Los Angeles, CA, USA, Tech. Rep. TR-0200 (4230-46)-3, Nov. 1988.
S. Albeaik, T. Wu, G. Vurimi, F.-C. Chou, X.-Y. Lu, and A. M. Bayen, “Deep truck cruise control,” Control Engineering Practice, vol. 121, p. 105026, Apr. 2022, doi: 10.1016/j.conengprac.2021.105026.
Q. Wang, J. Cheng, and H. Zhang, “Policy Gradient Reinforcement Learning Method for Backward Motion Control of Tractor-Trailer Mobile Robot,” in Proceedings of the 11th International Conference on 141 Computer Engineering and Networks, vol. 808, Q. Liu, X. Liu, B. Chen, Y. Zhang, and J. Peng, Eds. Singapore: Springer Nature Singapore, 2022, pp. 303–311. doi: 10.1007/978-981-16-6554-7_35.
M. Jaritz, R. de Charette, M. Toromanoff, E. Perot, and F. Nashashibi, “End-to-End Race Driving with Deep Reinforcement Learning,” in 2018 IEEE International Conference on Robotics and Automation (ICRA), Brisbane, QLD, May 2018, pp. 2070–2075. doi: 10.1109/ICRA.2018.8460934.
C. Altafini, A. Speranzon, and B. Wahlberg, “A feedback control scheme for reversing a truck and trailer vehicle,” IEEE Trans. Robot. Automat., vol. 17, no. 6, pp. 915–922, Dec. 2001, doi: 10.1109/70.976025.
C. Altafini, “Controllability and singularities in the n-trailer system with kingpin hitching,” IFAC Proceedings Volumes, vol. 32, no. 2, pp. 2169–2174, Jul. 1999, doi: 10.1016/S1474-6670(17)56368-5.
Arulkumaran, Kai, et al. "Deep reinforcement learning: A brief survey." IEEE Signal Processing Magazine 34.6 (2017): 26-38.
J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Proximal Policy Optimization Algorithms.” arXiv, Aug. 28, 2017. Accessed: Apr. 25, 2023. [Online]. Available: http://arxiv.org/abs/1707.06347
Williams, Ronald J. "Simple statistical gradient-following algorithms for connectionist reinforcement learning." Reinforcement learning (1992): 5-32.
Lee, Kyowoon, et al. "Deep reinforcement learning in continuous action spaces: a case study in the game of simulated curling." International conference on machine learning. PMLR, 2018.
Y. Yu, X. Si, C. Hu, and J. Zhang, “A Review of Recurrent Neural Networks: LSTM Cells and Network Architectures,” Neural Computation, vol. 31, no. 7, pp. 1235–1270, Jul. 2019
F. Sorge, “Motion and force analysis of slow-speed multi-trailer systems,” Proceedings of the Institution of Mechanical Engineers, Part C: Journal of Mechanical Engineering Science, vol. 233, no. 8, pp. 2615–2635, Apr. 2019, doi: 10.1177/0954406218792589.
X. Liu, A. K. Madhusudhanan, and D. Cebon, “Minimum Swept-Path Control for Autonomous Reversing of a Tractor Semi-Trailer,” IEEE Trans. Veh. Technol., vol. 68, no. 5, pp. 4367–4376, May 2019, doi: 10.1109/TVT.2019.2895513.
G. Lei and Y. Zheng, “Research on Cooperative Trajectory Planning Algorithm Based on Tractor-Trailer Wheeled Robot,” IEEE Access, vol. 10, pp. 64209–64221, 2022, doi: 10.1109/ACCESS.2021.3062392.
P. Polack, F. Altche, B. d’Andrea-Novel, and A. De La Fortelle, “The kinematic bicycle model: A consistent model for planning feasible trajectories for autonomous vehicles?,” in 2017 IEEE Intelligent Vehicles Symposium (IV), Los Angeles, CA, USA: IEEE, Jun. 2017, pp. 812–818. doi: 10.1109/IVS.2017.7995816.
K. McRae and P. Hetherington. Catastrophic interference is eliminated in pre-trained networks. In CogSci, 1993. 1
H. Xu, Z. Yan, J. Xuan, G. Zhang, and J. Lu, “Improving proximal policy optimization with alpha divergence,” Neurocomputing, vol. 534, pp. 94–105, May 2023, doi: 10.1016/j.neucom.2023.02.008.
J. Schulman, S. Levine, P. Moritz, M. I. Jordan, and P. Abbeel, “Trust Region Policy Optimization.” arXiv, Apr. 20, 2017. Accessed: Jun. 04, 2023. [Online]. Available: http://arxiv.org/abs/1502.05477
Y. Wang, H. He, and X. Tan, “Truly Proximal Policy Optimization” Uncertainty in Artificial Intelligence. PMLR, 2020.
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2023 Hao Yan, Mohamed A Zohdy, Abdelrahman Shaout, Amr Mahmoud

This work is licensed under a Creative Commons Attribution 4.0 International License.