- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper examines whether improvements in driving benchmark scores remain reliable indicators of better behavior when those scores are used as optimization targets. It separates evaluation into execution, measurement, component scoring, and aggregation, and studies how omitted behaviors, thresholded metrics, and controller transformations can hide changes in planner outputs.
Critical strengths
-
The controlled execution-interface comparison is the strongest contribution. The authors hold the planner checkpoints, scenes, scoring, and inference settings fixed while changing how trajectories are executed. The resulting reversals provide concrete evidence that conclusions about optimization gains can depend on the execution and feedback process, going beyond a general observation that benchmark metrics are imperfect.
-
The experimental analysis is extensive and reasonably careful. It combines multiple planner families with targeted interventions, paired uncertainty estimates, and counterexamples where optimization improves both scores and behavior. This supports a more credible and qualified conclusion than an analysis based only on selected failure cases.
Critical weaknesses
-
The novelty and primary takeaway could be clearer. The limitations of PDMS and the mismatch between benchmark scores and driving behavior are already recognized in autonomous driving research. Although the paper provides extensive experiments, the discussion moves among metric omission, thresholding, trajectory quality, and execution-interface effects without consistently distinguishing established limitations from new findings. This makes it difficult to identify the specific advance offered by the study.
-
The paper includes closed-loop experiments, but the main execution-interface comparison uses four-second, failure-selected scenarios with non-reactive traffic. These results do not establish whether the observed score–behavior mismatches persist over complete routes or affect route completion and task success. A clearer connection between the reported diagnostics and these longer-term outcomes would strengthen the conclusions.
-
Equation 3 already defines and uses implicit multiplication. Table 1 & 3 some numbers are not matching. The authors should clarify the numerical precision and explain these discrepancies.
Questions for authors
-
Which result do the authors consider the primary new finding beyond previously reported metric saturation and open-/closed-loop disagreement? Is the intended contribution chiefly the controlled execution-interface reversal, or a broader claim that the evaluated optimization methods reduce driving quality?
-
How should the increases in planned jitter, smoothing displacement, and executed jerk be interpreted in practical terms? What evidence distinguishes a meaningful deterioration in safety or comfort from an acceptable trade-off accompanying improvements in progress and road compliance?
4 comments
R: Do the increases in jitter, smoothing displacement and jerk reflect real deterioration, or an acceptable trade-off for progress and road compliance?
A: Thank you for raising this point. To test whether these diagnostics relate to real closed-loop driving, we compared Plan-R1 IL and RL within the same nuPlan val14 scene, both executed by nuPlan's LQR tracker in non-reactive traffic. We selected 369 scene pairs in which the two plans have similar acceleration peaks and path lengths, and split them into thirds by the RL − IL change in each diagnostic.
| RL − IL CLS | RL smoother | Similar | RL less smooth |
|---|---|---|---|
| Smoothing displacement | +0.034 | +0.026 | −0.001 |
| Heading jitter | +0.035 | +0.024 | 0.000 |
Where RL's plans become less smooth than IL's, RL's closed-loop advantage disappears. These diagnostics therefore relate to closed-loop outcomes, and the same-scene comparison shows whether a loss of smoothness is offset by gains in progress and road compliance.
R: Eq. 3 reuses with implicit multiplication, and some numbers in Tables 1 and 3 do not match.
A: Thank you for pointing out these three issues. We will correct the mismatches in Tables 1 and 3 by computing each change from the displayed values rather than from the unrounded means, so that all entries are consistent. We will also report all values to three decimals and label the EPDMS column in each table (Table 1: navtest rescoring of fixed outputs; Table 3: official two-stage navhard score). In Eq. 3, we will replace with , consistent with in Eq. 2, and add an explicit multiplication sign.
Thank the authors for the detailed response. These clarifications effectively address my questions, and I look forward to seeing these modifications incorporated into the revised manuscript. With my concerns resolved, I am raising my score to 3.
We thank the reviewer for the thoughtful engagement and for raising the score. The questions on the primary finding and its scope helped us state the contribution more precisely. In the revised version, we will distinguish established observations from our new findings in the Introduction and Conclusion, add the reactive, longer-horizon, and complete-route experiments, and correct Tables 1 and 3 and Eq. 3.
Thank you for your constructive feedback on the paper's contribution, the interpretation of the diagnostics, the evaluation setting, and numerical presentation. We address these points below.
R: What is the primary new finding (the execution-interface reversal or reduced driving quality), and does it hold beyond four-second, failure-selected, non-reactive scenarios, e.g., on complete routes?
A: Thank you for helping us clarify the primary new finding. Beyond previously reported metric saturation and open-/closed-loop disagreement, our primary finding is that the measured benefit of optimization is conditional on the execution interface: for the same IL–RL planner pair, changing how predicted trajectories are executed can substantially change the measured RL gain and, in some cases, reverse its sign. Our claim is therefore not that RL generally reduces driving quality. Rather, a reported optimization gain is not solely a property of the optimized planner; it is conditional on the execution and evaluation setting under which that gain is measured.
We also thank the reviewer for suggesting reactive traffic, longer horizons, and complete routes. We added reactive-traffic and 8-s results on WorldEngine, 15-s closed-loop results on nuPlan, and complete-route results on NavSafe-∞ (Bao et al., 2026). The table reports the RL − IL gain:
These additional results clarify the scope of the finding. First, the planner–executor reversal is not specific to non-reactive traffic: it persists with reactive traffic in WorldEngine. Second, it is not an artifact of the 4-s horizon: at 8 s, the RL gain under LQR nearly disappears (+0.006), whereas under playback RL falls further behind IL (−0.489). Third, executor dependence extends beyond WorldEngine: on 15-s nuPlan evaluation and complete-route NavSafe evaluation, the gain remains positive but changes substantially with the executor. Thus, the general finding is executor dependence of the optimization gain; sign reversal is a stronger case that occurs under some evaluation conditions. We will revise the paper to make the distinction between established observations and our new findings explicit. Our contributions are:
We will revise the Introduction and Conclusion to state this hierarchy explicitly and incorporate the new reactive, longer-horizon, and complete-route experiments.