acceptodds
Under review as a conference paper at ICLR 2027

Autonomous Traffic through Self-Play

Abstract

Traffic rules—lanes, prescribed travel directions, signals, and right-of-way—solve a coordination problem, designed around the limitations of human drivers. We ask what organization a population of autonomous vehicles would discover if these rules were not imposed, and how it would compare with rule-based autonomous traffic. Without rules, vehicles must converge on conventions for passing, merging, and intersection traversal that no one specifies, in continuous two-dimensional road space, with each vehicle acting independently based on local observations. To learn at the required scale, we introduce OpenDrive-GPU, a batched multi-agent driving simulator, and AutoZero, a self-play method combining spatial attention with phasic policy–value learning. Trained on several road networks from hundreds of billions of vehicle transitions without human demonstrations, a single shared policy exhibits recognizable crossing, yielding, merging, and passing behaviors. Evaluated on held-out road networks, rule-free autonomous traffic sustains 1.78 and 2.63 the throughput of the best-performing rule-based traffic at medium and high demand, respectively. While these results demonstrate the efficiency potential of self-organized autonomous traffic, they also reveal safety challenges: collisions and road departures increase under heavy demand, and throughput shows sensitivity to failure clearance. These findings identify safe coordination as a central open challenge for rule-free autonomous traffic.

Primary area:
applications to robotics, autonomy, planning
Keywords:
autonomous driving, self-play, multi-agent reinforcement learning, emergent behaviors, traffic simulation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_YC34OpenReview ↗
Rating
4
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper introduces OpenDrive-GPU, a batched multi-agent driving simulator, and AutoZero, a self-play method combining spatial attention with phasic policy–value learning. The experiments were run on several road networks with hundreds of billions of vehicle transitions without human demonstrations. A single shared policy exhibits recognizable crossing, yielding, merging, and passing behaviors. Evaluated on held-out road networks, rule-free autonomous traffic sustains 1.78× and 2.63× the throughput of the best-performing rule-based traffic at medium and high demand, respectively.

The results demonstrate the efficiency potential of self-organized autonomous traffic, but they also reveal safety challenges: collisions and road departures increase under heavy demand, and throughput shows sensitivity to failure clearance. They also identify safe coordination as a central open challenge for rule-free autonomous traffic.

The main article is composed of the introduction with motivation. Then, it contains a related work section and a description of the methodology. These sections are followed by the evaluation and analysis. The conclusions are followed by the AI use, ethics, and reproducibility statements. The main section is followed by a comprehensive appendix.

Critical strengths

This is a well-written paper, it contains strong efficiency results, reported together with safety costs. The paper seems to be original: it answers a new, clearly framed research question, studied at the network scale. I also appreciate that the paper introduces OpenDrive-GPU, a batched multi-agent driving simulator that enables large-scale training across continuous road networks. It is a scalable infrastructure that others can reuse. I also appreciate the fact that the authors included a discussion of limitations and a comprehensive appendix.

Critical weaknesses

I did not identify any significant weaknesses.

Questions for authors

I do not have any specific questions to authors.

1 comment

Thank you for your review and for your appreciation of our work. We will continue to improve OpenDrive-GPU, and we hope that it, together with this study, can contribute to research on autonomous traffic and to the broader community. In the revision, we will make the paper easier to follow and add further experiments and analysis to make it more comprehensive.

Reviewer_CxQbOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether autonomous vehicles can develop coordinated traffic behaviors without predefined lanes, traffic signals, right-of-way rules, or centralized control. The authors introduce OpenDrive-GPU, a GPU-based traffic simulator, and AutoZero, a multi-agent reinforcement learning framework trained through large-scale self-play. The results demonstrate substantial throughput improvements over conventional signal-based traffic control on a held-out grid network, although these gains are accompanied by increased collision rates.

Critical strengths
  1. Investigating whether coordinated traffic behavior can emerge without predefined traffic rules is an interesting direction with potentially significant implications for autonomous transportation.

  2. The GPU simulator enables extensive multi-agent training, and the authors provide detailed experimental settings, statistical analyses, and candid discussions of failure cases, including collisions and gridlock.

Critical weaknesses
  1. The headline throughput improvements are obtained under different simulation dynamics, information availability, and safety conditions. In particular, AutoZero benefits from shared neighboring-route information and incurs collisions, whereas the signal-based baselines operate without collisions. Consequently, the reported throughput advantage does not establish superiority under comparable safety and information constraints. Stronger autonomous traffic baselines are also missing at the network level.

  2. The rule-intervention experiments impose constraints on an already trained policy rather than comparing policies trained under different rule settings. Furthermore, the main policy uses a single training seed, and the observed directional preferences may partly reflect training-map geometry. These results do not yet establish that removing traffic rules is responsible for the performance gains or that robust traffic conventions have emerged.

Questions for authors
  1. How does AutoZero's throughput change when the collision penalty is increased to achieve safety levels comparable to the baselines?
  2. Can the authors compare against reservation-based or cooperative autonomous intersection management methods under matched simulation and information conditions?
  3. Would policies trained separately under lane, direction, or signal constraints support the same conclusions as the current rule-intervention experiments?
  4. Do independently trained policies with different random seeds converge to consistent or distinct traffic conventions?
5 comments
AuthorsOpenReview ↗

Response to Reviewer CxQb

Thank you for the specific and constructive review. Following your suggestions, we fine-tuned AutoZero with higher failure penalties, ran all network baselines in OpenDrive-GPU with two added cooperative methods, and trained 4 additional seeds. We report the results below, and the revision will state the scope of each comparison more precisely. W1 and W2 refer to your two weaknesses.

AuthorsOpenReview ↗

Throughput at reduced failure rates (W1, Q1)

To answer your question, we fine-tuned (FT) the AutoZero policy reported in the paper on the evaluation network with the original reward and with collision and off-road penalties raised to 2× and 4×. Each fine-tuning run used the same budget of 10B transitions. All results are from OpenDrive-GPU, where the signal baselines now share vehicle dynamics, collision checks and failure handling with AutoZero, so the ratios differ from those in the original manuscript (see response to Q2 below). The results are as follows:

  • Higher penalties reduced failures without lowering throughput. At high demand, 4× penalties reduced collisions from 0.87% to 0.35% and road departures from 0.27% to 0.01%, while throughput stayed at 1.99× that of the best signal baseline (1.97× before fine-tuning). Medium demand showed the same pattern. At low demand, all fine-tuned policies completed more than 99.9% of trips, and their throughput was within 3% of the best signal baseline.
  • Fine-tuning alone also helped. With the original reward, collisions at high demand fell from 0.87% to 0.58%; higher penalties reduced them further.

Failures did not reach the zero level of the signal baselines, but the throughput advantage persisted as failures decreased, so it does not depend mainly on the original failure rate.

DemandMethodThroughput (veh/h)Completion (%)Collision (%)Off-road (%)
MediumBest signal baseline6,84289.840.000.00
AutoZero9,53499.360.380.25
AutoZero (FT), original penalties9,55899.750.240.01
AutoZero (FT), 2× penalties9,55299.730.260.01
AutoZero (FT), 4× penalties9,57599.900.090.01
HighBest signal baseline7,22767.810.000.00
AutoZero14,25398.850.870.27
AutoZero (FT), original penalties14,34799.370.580.05
AutoZero (FT), 2× penalties14,36599.460.530.01
AutoZero (FT), 4× penalties14,38799.640.350.01

Safety is a long-standing challenge for autonomous transportation, both in practice and in research [1, 2]. The failures of rule-free traffic must likewise be reduced before its efficiency can be relied on. This connects to the motivation of our study: understanding how autonomous vehicles coordinate without rules can inform how new rules could be designed to improve safety while keeping much of the efficiency. We plan to pursue this direction in future work.

References

[1] Kalra and Paddock. Driving to safety: How many miles of driving would it take to demonstrate autonomous vehicle reliability? Transp. Res. Part A, 2016.

[2] Shalev-Shwartz et al. Safe, multi-agent, reinforcement learning for autonomous driving. arXiv:1610.03295, 2016.

AuthorsOpenReview ↗

Baseline comparison under matched simulation conditions (W1, Q2)

We ran all network baselines in OpenDrive-GPU, so that they share vehicle dynamics, collision checks and failure handling with AutoZero. Besides the three signal-based baselines, we added two cooperative methods: the reservation-based intersection manager with first-come, first-served ordering (AIM-FCFS) [3], and OBS-KATS [4], a cooperative driving method based on multi-agent path finding. Both are centralized: the intersection manager knows the routes and kinematic states of all vehicles approaching the intersection, so at intersections they have at least as much information as AutoZero. To show these methods at their best, we adapted them to the vehicle dynamics, added safety constraints to avoid collisions, and tuned their parameters, including a fine-grained 1 m grid resolution for space reservation. The results are as follows:

  • AutoZero achieved the highest throughput at all three demand levels. At medium and high demand, it reached 1.39× and 1.97× that of the strongest baseline, the actuated signal controller. These ratios are lower than the 1.78× and 2.63× in the original manuscript because the baselines previously ran in SUMO; in OpenDrive-GPU, they reached higher throughput, while AutoZero's results are unchanged. At low demand, all methods were limited by demand, and AutoZero exceeded Actuated by about 2% (4,798 versus 4,694 veh/h).
  • The cooperative methods showed no clear advantage over the signal-based baselines on the network. At high demand, they reached 6,910 and 6,694 veh/h, compared with 7,227 veh/h for the actuated controller. Both methods were designed for a single intersection and do not account for upstream and downstream intersections. Both also assign crossing order by reserving a discretized junction space, so they cannot use road space as flexibly as AutoZero. In addition, demand in our scenarios is asymmetric across approaches. Prior work has also found that FCFS reservations can cause longer delays than signals in such settings: where a major road meets a minor road, FCFS alternates between them instead of favoring the major road, and on networks it can break the coordination between neighboring intersections [5].
DemandMethodThroughput (veh/h)Completion (%)
MediumActuated6,84289.84
Max-Pressure6,03482.71
MPLight6,33684.65
AIM-FCFS6,43083.58
OBS-KATS6,32885.02
AutoZero9,53499.36
HighActuated7,22767.81
Max-Pressure6,30659.47
MPLight6,30259.34
AIM-FCFS6,91065.92
OBS-KATS6,69464.39
AutoZero14,25398.85

References

[3] Dresner and Stone. A multiagent approach to autonomous intersection management. JAIR, 2008.

[4] Yan et al. Multi-agent path finding for cooperative autonomous driving. ICRA, 2024.

[5] Levin, Boyles and Patel. Paradoxes of reservation-based intersection controls in traffic networks. Transp. Res. Part A, 2016.

Reviewer_h1H2OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This work focuses on self-play coordination without traffic rules. It developed a batched simulator OpenDrive-GPU that enables large-scale batched GPU simulation and a modified version of PPG to learn coordination behavior with axuliary phasic policy-value delay. Experiments shows the self-play can learn interaction beahviors such as yielding and merging without explicit reward design and better throughput compare to other baselines such as MPC.

Critical strengths

The speed of OpenDrive-GPU is interesting, 22.93 Million agents/step in one RTX 3090

Critical weaknesses

The main contribution in this work remain unclear, why is learning coordination behavior without reward design is important?

Coordination behavior is also shaped by reward design and observation. If partial observability, such as occlusion are introudced, the final beahvior would also be different. OpenDrive-GPU does not quantify whether the emgering behavior(yielding or merging) is realistic,

Evlaution are insufficient, baselines are non-self play based method and mostly rule-based baseline. Authors should compare to PufferDrive, or other self-play methods on existing benchmarks such as WOSAC to have a quantitative metrics other than throughput.

Questions for authors
  • Can the author describe implmentation detials on how it achieve much faster throughput than PufferDrive/GPU drive? What are the key components for better training throughput
  • Why is learning coordination without traffic rules important? In the near term, autonomous Driving will adapt to the real-world traffic rules, to be predictable and adaptable to other drivers.
  • Additional evaluations and comparisons with self-play baselines, such as evaluations using closed-loop planning metrics or simulation-based agent metrics (e.g., WOSAC), would help provide a more rigorous assessment of the proposed method.
8 comments
AuthorsOpenReview ↗

Response to Reviewer h1H2

Thank you for your review. We would like to clarify a possible misunderstanding about the aim of the paper. It studies a question different from that of conventional autonomous-driving research: it concerns autonomous traffic rather than individual autonomous vehicles. Typical autonomous-driving research asks how vehicles can drive well within today's traffic system: a vehicle stops at a red light and keeps to its lane, and benchmarks such as WOSAC [1] measure how closely it matches human drivers. We instead ask what the traffic system itself could look like when all vehicles are autonomous, and the rules designed for human drivers are not imposed. This matters because these rules are built around human limitations that autonomous vehicles do not share.

Prior work has begun to study this question, but typically at a single intersection or road segment, for example, signal-free intersection management [2–4] and lane-free traffic [5–7]. In contrast, we remove all of these rules and study their effect on road networks. Understanding how a population of autonomous vehicles coordinates without rules can inform how traffic rules for autonomous vehicles should be designed. The other reviewers read the paper this way: Reviewer YC34 described it as answering "a new, clearly framed research question, studied at the network scale", and Reviewer CxQb as "an interesting direction with potentially significant implications for autonomous transportation".

Below, W1–W3 refer to your weaknesses in the order listed.

References

[1] Montali et al. The Waymo Open Sim Agents Challenge. NeurIPS Datasets and Benchmarks, 2023.

[2] Dresner and Stone. A multiagent approach to autonomous intersection management. JAIR, 2008.

[3] Xu et al. Comparison of cooperative driving strategies for CAVs at signal-free intersections. IEEE T-ITS, 2022.

[4] Yan et al. Multi-agent path finding for cooperative autonomous driving. ICRA, 2024.

[5] Papageorgiou et al. Lane-free artificial-fluid concept for vehicular traffic. Proc. IEEE, 2021.

[6] Li et al. Lane-free autonomous intersection management: A batch-processing framework integrating reservation-based and planning-based methods. ICRA, 2021.

[7] Naderi et al. Lane-free signal-free intersection crossing via model predictive control. Control Eng. Pract., 2025.

AuthorsOpenReview ↗

Motivation for studying autonomous traffic (W1, Q2)

Our response above explains why we study traffic without rules. On your point about the near term, we agree that autonomous vehicles must follow existing rules to remain predictable to human drivers, as long as they share the road with them. Our question concerns the longer term, when all vehicles are autonomous. Rules designed around human limitations may then no longer be needed, or may need to be redesigned for autonomous vehicles.

AuthorsOpenReview ↗

Reward design and observation assumptions (W1, W2)

Reward and observation do shape coordination behavior. In conventional autonomous driving, rewards encode traffic rules designed for human drivers. For example, the policy of [8] is rewarded for driving in the designated direction and staying centered in its lane, and penalized for crossing a stop line at a red light; the coordination it learns is therefore guided by these rules. Our question is what coordination autonomous vehicles would adopt, and with what effect, once these rules are removed. We therefore restrict the reward to goals, failures, progress and time, and avoid terms that encode human-designed rules. In addition, we assume ideal communication in this study: neighbor states and routes are shared reliably and without delay (Sections 3 and 4.4). How robust the learned coordination is under occlusion, sensing noise and communication errors deserves a separate study.

References

[8] Cusumano-Towner et al. Robust autonomy emerges from self-play. ICML, 2025.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Overall Response and Summary of Revision (1/2)

We thank the reviewers for their feedback. Reviewers found the research question new, clearly framed and of potential significance for autonomous transportation. They also raised questions about the evaluation and baselines, the relation between traffic throughput and failure rates, training seeds and the scope of the rule experiments. We respond to each review individually; here we summarize the main changes in the revised version we have uploaded:

  • Matched network comparison (Section 4.2.2, Table 2, Appendices F.1–F.2): all network baselines now run in OpenDrive-GPU with the same vehicle dynamics, collision checks and road-boundary checks as AutoZero. We added two cooperative methods, AIM-FCFS and OBS-KATS, which use the current states and paths of vehicles at each junction. AutoZero achieves the highest throughput at all three demand levels, 1.02×, 1.39× and 1.97× that of the strongest baseline. These ratios replace the 1.78× and 2.63× in the original manuscript, whose baselines ran in SUMO.
  • Traffic throughput and failure rates (Sections 4.2.2 and 4.3, Table 3, Appendix F.3): fine-tuning AutoZero with higher collision and off-road penalties reduced high-demand collisions from 0.87% to 0.35% and road departures from 0.27% to 0.01%, while throughput reached 1.99× that of the strongest baseline. The demand-stress and failure-clearance experiments now also include the fine-tuned policy.
  • Training seeds (Section 4.1, Appendix I): we trained 4 additional policies with different random seeds. Including the policy in the paper, 3 of the 5 prefer the left side and 2 prefer the right, and all exhibit the crossing, yielding, passing, merging and following behaviors.
  • Limitations (Section 4.4): the limitations now discuss separately the effect of individual rules, the remaining safety gap, and the assumptions of all-autonomous traffic and reliable communication. The rule-intervention experiments are described as tests of a fixed policy in single scenarios, and training under each rule setting on road networks is left to future work.
  • Presentation: we updated the text throughout to reflect the new results, and improved the writing for clarity. In particular, we clarified that the simulator benchmark measures simulation throughput rather than training throughput.
AuthorsOpenReview ↗

Overall Response and Summary of Revision (2/2)

We thank the reviewers again for their feedback. In addition to the changes summarized in (1/2), we have also extended the behavior analysis:

  • Coordination among agents (Section 4.1, new Figure 3, Appendix M): building on the emergent behaviors in the original paper, we further analyzed how agents coordinate without prescribed rules, focusing on the passing-side convention, right-of-way and attention.
  • Comparison with human drivers (Appendix N): we compared driving metrics such as speed, acceleration and time to collision with human trajectories from the INTERACTION and NGSIM datasets.

Together, these changes do not alter the research question or the main conclusions.

As the discussion period ends soon, we would be grateful if the reviewers could look at our responses and the revised paper and let us know whether they address the questions raised. We are happy to clarify anything further.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.