- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper proposes a constrained adversarial self-play framework for autonomous driving. A KL budget around a behavioral prior controls adversarial strength, while CDF-based risk estimates guide learning to prioritize risk reduction or task performance depending on the state. The paper analyzes feasible-region expansion and safety transfer across adversarial strengths, and evaluates attack-realism trade-offs and driving performance in PufferDrive using Waymo Open Motion Dataset scenarios.
Critical strengths
-
Strong empirical evaluation. Experiments on 10,000 training and 1,000 held-out scenarios cover multiple baselines and nominal, adversarial, and cross-strength settings. Co-SPAR improves safe task completion by over 10 percentage point over the strongest baseline, supporting its practical value.
-
Extensive theoretical analysis. Under the stated assumptions, the paper establishes feasible-region expansion, CDF convergence, and safety-certificate transfer across adversarial strengths. These results connect the adversary constraints to safety evaluation and protagonist optimization.
-
The studied problem is important, and the overall framework is reasonable.
Critical weaknesses
-
Gap between theory and implementation. The theory assumes exact policy iteration and worst-case adversary optimization, while the implementation uses learned CDF critics, PPO, empirical average KL control, and decentralized updates. It is unclear how these approximations affect feasible-region estimates and safety preservation, or which guarantees apply to the trained policies.
-
Unclear multi-agent credit assignment. Overlapping neighborhoods can assign the same protagonist violation to several adversaries without distinguishing their contributions. The paper needs stronger justification or targeted experiments showing that these local updates support the joint adversarial objective.
-
Limited distinction from prior work. Adversarial self-play and ego-adversary co-evolution have been studied in prior work, such as ADV-0. Beyond cross-strength safety theory, the differences in optimization objectives and policy updates remain unclear.
Questions for authors
See weakness.
5 comments
Thank you again for your detailed feedback. Please let us know if any concerns remain after our response or if further clarification would be helpful.
We appreciate your time and would be grateful to hear your thoughts on the explanations provided in our response.
We sincerely appreciate your detailed and constructive feedback. Below are our responses to your comments.
W1 (Safety guarantees for learned policies)
Co-SPAR’s safety certificate applies to a learned protagonist at states where its true robust CDF satisfies the feasibility threshold. In training, a learned adversary with average KL control approximates the worst admissible adversary under the state-wise KL constraint. The critic’s estimate determines which update a state receives. Near the feasibility threshold, an estimation error can switch the update from task improvement to risk reduction or vice versa. Our monotone expansion result applies to the exact region-wise policy improvement step that the decentralized PPO updates approximate.
Analyzing an idealized policy update and approximating it in training has precedent in safe RL. Feasible Policy Iteration [1] assumes accurate CDF estimates in its analysis and learns the CDF in continuous state spaces. Constrained Policy Optimization [2] derives a constrained update and implements it using a local approximation and line search.
[1] Yujie Yang et al. Feasible Policy Iteration for Safe Reinforcement Learning. arXiv:2304.08845, 2023.
[2] Joshua Achiam et al. Constrained Policy Optimization. ICML, 2017.
W2 (Overlapping adversary neighborhoods)
Two adversaries may observe the same violation, but under a factorized joint policy each adversary’s expected update depends on how its own actions affect the expected team return. Consider a vehicle alongside the protagonist just before another vehicle cuts in. If holding its position blocks the protagonist’s escape route while yielding leaves it open, a subsequent collision in both vehicles’ neighborhoods can provide a learning signal for that earlier choice. If the expected team return is the same for every action available to the vehicle alongside the protagonist, its expected gradient contribution at that observation is zero.
In training, each adversary uses a local CDF based on the first protagonist violation in its neighborhood. Using this local CDF in place of the team-wide signal relies on the neighborhood sufficiency assumption in Appendix C.4 (Assumption 2), under which the substitution preserves that adversary’s expected policy update. Under the independent update approximation, the per-agent steps taken from the same joint rollout together approximate a joint policy update.
W3 (Objectives and policy updates in ASP, ADV-0, and Co-SPAR)
Optimization objectives. ADV-0 formulates a zero-sum game over ego return. Its adversary learns a distribution favoring low-return traffic trajectories, and the ego maximizes return under that distribution. ASP trains a teacher to challenge the student while demonstrating a collision-free solution to the same scenario. In Co-SPAR, the adversary maximizes the CDF over a prior-constrained policy class. The resulting robust CDF identifies where the current protagonist is safe against every admissible adversary, while task return remains a separate protagonist objective.
Policy updates. In ADV-0, updating the adversary changes the traffic distribution for the next ego-return update. In Co-SPAR, evaluating the adversary also determines which objective the protagonist pursues at each state. On joint rollouts, the protagonist uses the estimated robust feasible region to improve task return with a CDF penalty inside the region and to reduce risk outside it.
At fixed β, each exact update preserves the current robust feasible region even when the worst admissible adversary is recomputed for the updated protagonist (Theorem 1). On that same region, task value is nondecreasing against the adversary used to compute the update (Theorem 3).
We added this comparison to the Related Work section.