针对无信号交叉口强交互环境下自动驾驶车辆左转的控制问题,基于规则的传统方法适应性较差,而强化学习(reinforcement learning, RL)方法在复杂环境下难以收敛,为此,该文提出了一种基于改进软演员-评论家模型的左转控制算法。首先,设计了一种混合策略训练框架,融合基于规则的改进Krauss跟驰模型与RL策略,并进行交替训练,以解决RL模型在复杂场景中探索效率较低的问题;其次,通过细粒度评价动作风险,构建了兼顾安全和效率的多目标奖励函数,从而降低了车辆碰撞率;最后,考虑到左转任务具有长时序特征,提出了基于任务分解的分阶段经验回放缓冲区机制,有效提升了关键驾驶阶段的样本利用率,并进一步提升了训练速率。结果表明:该文所提方法能显著加速模型训练进程,收敛速度相较于RL模型提升约50%,且将碰撞率降至5%以下,在通行效率和安全性上均优于RL模型。该文研究结果可为左转车辆的控制算法设计和RL收敛速度的提升提供参考。
Abstract
[Objective] Autonomous driving faces considerable challenges in scenarios such as unsignalized intersections due to the highly dynamic interactions among traffic participants and the unpredictability of other drivers’ intentions. Control algorithms for these scenarios need to ensure safety and real-time responsiveness and can generally be categorized into traditional model-based approaches, including rule-based and optimization-based methods, and learning-based approaches. Although traditional model-based methods are interpretable, they typically require scenario-specific modeling and careful manual design, making them less flexible and difficult to generalize across different environments. Consequently, learning-based methods have emerged as a promising alternative for autonomous driving control. In particular, deep reinforcement learning (DRL) enables agents to optimize their policies through continuous interaction with the environment. However, in complex traffic scenarios, DRL models may suffer from insufficient exploration and slow convergence. Furthermore, among the various interaction patterns at unsignalized intersections, conflict resolution between left-turning and through vehicles is one of the most representative scenarios because it involves strong vehicle interactions and significant decision-making uncertainty. Existing studies have rarely exploited the stage-dependent characteristics of the left-turn process, such as entering, gaming, and merging, even though such structural information may be valuable for vehicle control. To address these issues, this paper proposes an improved left-turn control algorithm based on the soft actor-critic (SAC) framework to balance traffic efficiency and safety. In addition, several training strategies are introduced to accelerate convergence. [Methods] This study developed a reinforcement learning (RL) framework with several improvements. First, a mixed strategy training (MST)-SAC framework was constructed, which combines a rule-based model (the Krauss model) with the RL policy. Experience data generated by both policies were stored in a shared experience buffer, enabling mixed sampling. The MST-SAC addressed the cold-start problem and mitigated low exploration efficiency in complex scenarios. Second, a penalty function based on time-to-collision was constructed from the perspectives of vehicle state and action, and a multiobjective reward function was designed to account for safety, comfort, efficiency, and task completion. Third, a phased experience replay buffer was designed to accommodate the long-sequence nature of left-turn tasks. This buffer relied on a new metric, the arrival middle rate (AMR), which reflected the agent’s real-time progress. During training, the sampling ratio was adjusted dynamically according to AMR. As a result, the model specifically enhanced learning in its weakest driving phases. [Results] A four-way, two-lane unsignalized intersection was constructed in SUMO. The proposed model achieved the best overall control performance in our experiments. In terms of convergence, the mixed strategy and phased learning accelerated training considerably. The convergence speed increased by approximately 50.0% compared with baseline models, and the proposed model demonstrated high stability during training. Under normal traffic flow (25-35 veh/min), the model achieved a 96.3% success rate. It considerably outperformed standard SAC and model predictive control. The model also exhibited strong robustness under congested traffic conditions (35-45 veh/min). In this scenario, the success rate remained at 93.3%, a decrease of 3 percentage points compared to normal traffic conditions. The improved model also reduced the average delay, waiting time, and number of stops for other vehicles. In addition, a sensitivity analysis confirmed the model’s reliability when the initial speed was below 5 m/s. Overall, the model showed strong adaptability to random and heterogeneous traffic conditions. [Conclusions] This study addresses the problem of left-turn vehicle control at unsignalized intersections by proposing an improved SAC-based RL algorithm that includes MST-SAC, a safety-oriented reward function, and a phased replay buffer. From the experimental results, the improved model outperforms the baseline models in most metrics, including training speed, success rate, collision rate, total delay, and robustness. The proposed methods facilitate the practical application of DRL in highly complex traffic scenarios.
关键词
自动驾驶 /
控制策略 /
强化学习 /
无信号交叉口
Key words
autonomous driving /
control strategy /
reinforcement learning /
unsignalized intersection
{{custom_sec.title}}
{{custom_sec.title}}
{{custom_sec.content}}
参考文献
[1] ROSEN H E, BARI I, PAICHADZE N, et al. Global road safety 2010-18: An analysis of global status reports[J]. Injury, 2025, 56(6): 110266.
[2] CHEN C, RICKERT M, KNOLL A. Combining task and motion planning for intersection assistance systems[C]// 2016 IEEE Intelligent Vehicles Symposium (IV). Gothenburg, Sweden: IEEE, 2016: 1242-1247.
[3] LIU J X, LUO Y G, XIONG H, et al. An integrated approach to probabilistic vehicle trajectory prediction via driver characteristic and intention estimation[C]// 2019 IEEE Intelligent Transportation Systems Conference (ITSC). Auckland, New Zealand: IEEE, 2019: 3526-3532.
[4] LI Z R, GONG J W, LU C, et al. Interactive behavior prediction for heterogeneous traffic participants in the urban road: A graph- neural-network-based multitask learning framework[J]. IEEE/ ASME Transactions on Mechatronics, 2021, 26(3): 1339-1349.
[5] ZHANG H J, ZHAO J, WANG R, et al. Multi-objective reinforcement learning algorithm and its application in drive system[C]// 200834th Annual Conference of IEEE Industrial Electronics. Orlando, USA: IEEE, 2008: 274-279.
[6] ARADI S. Survey of deep reinforcement learning for motion planning of autonomous vehicles[J]. IEEE Transactions on Intelligent Transportation Systems, 2022, 23(2): 740-759.
[7] DE BRUIN T, KOBER J, TUYLS K, et al. Integrating state representation learning into deep reinforcement learning[J]. IEEE Robotics and Automation Letters, 2018, 3(3): 1394-1401.
[8] CHEN J H, SHU T X, LI T, et al. Deep reinforced learning tree for spatiotemporal monitoring with mobile robotic wireless sensor networks[J]. IEEE Transactions on Systems, Man, and Cybernetics: Systems, 2020, 50(11): 4197-4211.
[9] NAIR A, MCGREW B, ANDRYCHOWICZ M, et al. Overcoming exploration in reinforcement learning with demonstrations[C]// 2018 IEEE International Conference on Robotics and Automation (ICRA). Brisbane, Australia: IEEE, 2018: 6292-6299.
[10] WANG F, ZHOU B, CHEN K, et al. Intervention aided reinforcement learning for safe and practical policy optimization in navigation[C]// Proceedings of the 2ndAnnual Conference on Robot Learning. Zürich, Switzerland: PMLR, 2018: 410-421.
[11] WU J D, HUANG Z Y, HU Z X, et al. Toward human-in-the-loop AI: Enhancing deep reinforcement learning via real-time human guidance for autonomous driving[J]. Engineering, 2023, 21: 75-91.
[12] HUANG W H, LIU H C, HUANG Z Y, et al. Safety-aware human-in-the-loop reinforcement learning with shared control for autonomous driving[J]. IEEE Transactions on Intelligent Transportation Systems, 2024, 25(11): 16181-16192.
[13] KAHN G, ABBEEL P, LEVINE S. LaND: Learning to navigate from disengagements[J]. IEEE Robotics and Automation Letters, 2021, 6(2): 1872-1879.
[14] PENG Z H, LI Q Y, LIU C X, et al. Safe driving via expert guided policy optimization[C]// Proceedings of the 5th Conference on Robot Learning. London, UK: PMLR, 2022: 1554-1563.
[15] SCHAUL T, QUAN J, ANTONOGLOU I, et al. Prioritized experience replay[C]// Proceedings of the 4th International Conference on Learning Representations. San Juan, Puerto Rico: ICLR, 2016: 1-21.
[16] ISELE D, COSGUN A. Selective experience replay for lifelong learning[C]// Proceedings of the 32nd AAAI Conference on Artificial Intelligence. New Orleans, USA: AAAI, 2018: 3302-3309.
[17] XIAO W X, YANG Y Y, MU X Y, et al. Decision-making for autonomous vehicles in random task scenarios at unsignalized intersection using deep reinforcement learning[J]. IEEE Transactions on Vehicular Technology, 2024, 73(6): 7812-7825.
[18] ZHANG J C, ZHANG Y S, YAO E J. A new framework for traffic conflict identification by incorporating predicted high- resolution trajectory and vehicle profiles in a CAV context[J]. Transportation Research Record: Journal of the Transportation Research Board, 2025, 2679(11): 445-462.
[19] POLACK P, ALTCHÉ F, D’ANDREA-NOVEL B, et al. The kinematic bicycle model: A consistent model for planning feasible trajectories for autonomous vehicles?[C]// 2017 IEEE Intelligent Vehicles Symposium (IV). Los Angeles, USA: IEEE, 2017: 812-818.
[20] KRAUSS S. Microscopic modeling of traffic flow: Investigation of collision free vehicle dynamics[R]. Berlin: Deutsches Zentrum fuer Luft-und Raumfahrt, 1998.
[21] YANG K, LI S, CHEN Y L, et al. Towards safe decision-making for autonomous vehicles at unsignalized intersections[J]. IEEE Transactions on Vehicular Technology, 2025, 74(3): 3830-3842.
[22] LUO J P, ZHANG T T, HAO R, et al. Real-time cooperative vehicle coordination at unsignalized road intersections[J]. IEEE Transactions on Intelligent Transportation Systems, 2023, 24(5): 5390-5405.
[23] AL-SHARMAN M, DEMPSTER R, DAOUD M A, et al. Self-learned autonomous driving at unsignalized intersections: A hierarchical reinforced learning approach for feasible decision- making[J]. IEEE Transactions on Intelligent Transportation Systems, 2023, 24(11): 12345-12356.
[24] 蒋明智, 吴天昊, 张琳. 基于深度强化学习的无信号交叉口车辆协同控制算法[J]. 交通运输工程与信息学报, 2022, 20(2): 14-24. JIANG M Z, WU T H, ZHANG L. Deep reinforcement learning based vehicular cooperative control algorithm at signal-free intersection[J]. Journal of Transportation Engineering and Information, 2022, 20(2): 14-24. (in Chinese)
基金
国家重点研发计划项目(2023YFB4302703)