- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2(+1)
Read review
Paper summary
Planner-Breaker trains a shared adversarial policy to expose failures of a fixed driving planner. Its reward favors collisions that could have been anticipated and avoided by braking. The method is evaluated against five planners, with additional transfer experiments on WOMD.
Critical strengths
The focus on avoidable and foreseeable collisions is useful for planner evaluation. The reusable policy avoids per-scenario optimization, and the reward and post-collision continuation ablations help explain the training design.
Critical weaknesses
-
KING and ReGentS are evaluated over 10 seconds, while Planner-Breaker can run for up to 256 seconds. Table 11 reports a median collision time of 10.5 seconds, so this difference is consequential. Results within a common evaluation horizon are needed to establish the size of the improvement.
-
The proposed criterion is both a central contribution and the main evaluation measure. However, it is unclear whether its counterfactual braking maneuver is executable under the constraints of every tested planner. Appendix A.4 describes braking at the maximum deceleration, while Appendix B.1 defines jerk-limited actions for the learned policies. Matching the maximum deceleration alone does not establish matching stopping behavior. If the counterfactual bypasses these dynamics, some collisions may be labeled avoidable even though the tested policy could not execute the required maneuver. The counterfactual also fixes the other vehicles to their recorded motion and then uses constant-velocity extrapolation, while ignoring certain secondary collisions based on an assumed response from following vehicles. These are consequential modeling choices. The paper does not establish how often they change the collision labels. Similarly, foreseeability depends on the response time and geometric safety margins, without a sensitivity analysis or an independent assessment of the resulting labels. I would find the claims more convincing if the authors checked a representative set of failures using a dynamically feasible avoidance controller and reported agreement with the proposed criterion.
-
The main adversarial comparisons are limited to KING and ReGentS, both trajectory-optimization methods. The paper discusses related learned approaches but does not include a representative learned adversary in the experiments. SEAL is relevant because it generates reactive adversarial behavior, while STRIVE is particularly relevant to the claim that generated failures should admit a safe solution. These methods have different assumptions and need not all be reproduced, but the current selection leaves an important gap. A simpler RL adversary trained with the same collision criterion would also help distinguish the benefit of the criterion from that of the multi-agent training design. The reward ablations are useful, but do not resolve this question. The WOMD experiment does not close this gap: it compares against log replay and nominal traffic policies, without adversarial generation baselines. It therefore supports transfer to different road layouts and initial conditions, but provides limited evidence about comparative effectiveness in that setting.
-
The cross-planner results support specialization, but do not explain which planning weaknesses the adversaries exploit. I missed concrete examples showing the original interaction, the point at which the hazard becomes foreseeable, the last feasible braking time, and the collision-free counterfactual. Such examples are important for assessing the meaning of “informative” failures. Table 17 mainly describes behavior averaged over entire episodes. Similar average speeds and goal-reaching rates do not establish that the maneuvers immediately preceding a collision are plausible. The paper would benefit from collision-centered behavior analysis and a breakdown of failure types, showing whether the method discovers varied planning deficiencies or repeatedly exploits a small set of maneuvers.
I lean toward weak rejection. The motivation is sound, but the unequal evaluation horizons and insufficient validation of the failure criterion leave the main conclusions inadequately supported.
Questions for authors
Does the advantage over KING and ReGentS persist when all methods are evaluated within the same 10-second horizon?
How do the methods compare in computational cost and planner queries, accounting for both adversarial-policy training and subsequent scenario generation?
Does counterfactual braking obey the same execution constraints as the tested planners, particularly the jerk limits of SP and SP-Cond?
How sensitive are the collision labels and method rankings to the assumed reaction time, braking deceleration, and lateral safety margin?
The driving statistics summarize entire episodes. Do the adversaries still exhibit plausible behavior immediately before collisions, or do they exploit unrealistic coordinated maneuvers?
Statistics such as average speed and the number of times the target was reached indicate that, overall, the opposing vehicle still exhibited normal driving behavior. However, this does not sufficiently demonstrate that the critical seconds immediately preceding the collision aligned with real-world traffic patterns.
Thank you for the detailed response and the additional experiments, which address most of my concerns. One remaining question: in Table 28, the foreseeable rate of KING decreases as ρ grows, while that of Planner-Breaker peaks at the training value ρ = 1 s on several targets. It would be helpful if the authors could briefly explain this. A small manual check of the collision labels would also make the criterion more convincing. Overall, I maintain my score.