acceptodds
Under review as a conference paper at ICLR 2027

When the Score Becomes the Target: Rethinking Metric Validity in Autonomous Driving

Abstract

Driving benchmark scores are increasingly used not only for evaluation but also as optimization targets. This raises a fundamental question: do score gains remain reliable evidence of driving improvement once the score itself is optimized? We address this question by examining how the scoring process responds to changes in driving behavior and whether the resulting gains persist under repeated execution and replanning. We decompose the process into execution, measurement, subscore mapping, and aggregation. Controlled interventions reveal substantial behavioral changes that receive little score response because distinctions are omitted, thresholded, or attenuated between requested and executed motion. Closed-loop comparisons further show that optimization gains can reverse when the execution interface changes, demonstrating their dependence on how requests are executed and returned as feedback. Together, these findings connect the behavioral distinctions preserved by a metric to the conditions under which its gains transfer. Metric validity under optimization therefore requires examining both what the scoring process measures and how the optimized behavior is executed.

Primary area:
applications to robotics, autonomy, planning
Keywords:
End-to-end planning, Metric validity, Goodhart's law, Reinforcement learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_MGNEOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper examines whether improvements in driving benchmark scores remain reliable indicators of better behavior when those scores are used as optimization targets. It separates evaluation into execution, measurement, component scoring, and aggregation, and studies how omitted behaviors, thresholded metrics, and controller transformations can hide changes in planner outputs.

Critical strengths
  • The controlled execution-interface comparison is the strongest contribution. The authors hold the planner checkpoints, scenes, scoring, and inference settings fixed while changing how trajectories are executed. The resulting reversals provide concrete evidence that conclusions about optimization gains can depend on the execution and feedback process, going beyond a general observation that benchmark metrics are imperfect.

  • The experimental analysis is extensive and reasonably careful. It combines multiple planner families with targeted interventions, paired uncertainty estimates, and counterexamples where optimization improves both scores and behavior. This supports a more credible and qualified conclusion than an analysis based only on selected failure cases.

Critical weaknesses
  1. The novelty and primary takeaway could be clearer. The limitations of PDMS and the mismatch between benchmark scores and driving behavior are already recognized in autonomous driving research. Although the paper provides extensive experiments, the discussion moves among metric omission, thresholding, trajectory quality, and execution-interface effects without consistently distinguishing established limitations from new findings. This makes it difficult to identify the specific advance offered by the study.

  2. The paper includes closed-loop experiments, but the main execution-interface comparison uses four-second, failure-selected scenarios with non-reactive traffic. These results do not establish whether the observed score–behavior mismatches persist over complete routes or affect route completion and task success. A clearer connection between the reported diagnostics and these longer-term outcomes would strengthen the conclusions.

  3. Equation 3 already defines hkh_k and uses implicit multiplication. Table 1 & 3 some numbers are not matching. The authors should clarify the numerical precision and explain these discrepancies.

Questions for authors
  1. Which result do the authors consider the primary new finding beyond previously reported metric saturation and open-/closed-loop disagreement? Is the intended contribution chiefly the controlled execution-interface reversal, or a broader claim that the evaluated optimization methods reduce driving quality?

  2. How should the increases in planned jitter, smoothing displacement, and executed jerk be interpreted in practical terms? What evidence distinguishes a meaningful deterioration in safety or comfort from an acceptable trade-off accompanying improvements in progress and road compliance?

4 comments

Thank you for your constructive feedback on the paper's contribution, the interpretation of the diagnostics, the evaluation setting, and numerical presentation. We address these points below.

R: What is the primary new finding (the execution-interface reversal or reduced driving quality), and does it hold beyond four-second, failure-selected, non-reactive scenarios, e.g., on complete routes?

A: Thank you for helping us clarify the primary new finding. Beyond previously reported metric saturation and open-/closed-loop disagreement, our primary finding is that the measured benefit of optimization is conditional on the execution interface: for the same IL–RL planner pair, changing how predicted trajectories are executed can substantially change the measured RL gain and, in some cases, reverse its sign. Our claim is therefore not that RL generally reduces driving quality. Rather, a reported optimization gain is not solely a property of the optimized planner; it is conditional on the execution and evaluation setting under which that gain is measured.

We also thank the reviewer for suggesting reactive traffic, longer horizons, and complete routes. We added reactive-traffic and 8-s results on WorldEngine, 15-s closed-loop results on nuPlan, and complete-route results on NavSafe-∞ (Bao et al., 2026). The table reports the RL − IL gain:

SettingModelMetricLQRPlayback
WorldEngine, 4 sQwen-DrivePDMS+0.136−0.380
WorldEngine, 4 s, reactiveQwen-DrivePDMS+0.149−0.358
WorldEngine, 8 sQwen-DrivePDMS+0.006−0.489
nuPlan val14, 15 sPlan-R1CLS+0.034+0.138
NavSafe, complete routesReCogDrive-2BRoute completion (points)+5.8+13.3
NavSafe, complete routesReCogDrive-2BTask success (points)+3.3+15.0

These additional results clarify the scope of the finding. First, the planner–executor reversal is not specific to non-reactive traffic: it persists with reactive traffic in WorldEngine. Second, it is not an artifact of the 4-s horizon: at 8 s, the RL gain under LQR nearly disappears (+0.006), whereas under playback RL falls further behind IL (−0.489). Third, executor dependence extends beyond WorldEngine: on 15-s nuPlan evaluation and complete-route NavSafe evaluation, the gain remains positive but changes substantially with the executor. Thus, the general finding is executor dependence of the optimization gain; sign reversal is a stronger case that occurs under some evaluation conditions. We will revise the paper to make the distinction between established observations and our new findings explicit. Our contributions are:

  1. An empirical finding about the conditionality of optimization gains. A statement such as "RL improves the planner by X points" is often interpreted as a property of the optimized planner. Our results show that X is conditional on the evaluation setting: it can change substantially with the execution interface and horizon and, in some settings, can even change sign.
  2. A reusable diagnostic method. Section 3 decomposes the scoring process into execution, measurement, subscore mapping, and aggregation, and uses controlled behavioral interventions to test whether relevant behavioral distinctions survive this process. This identifies where behavioral distinctions can be lost or attenuated before interpreting an observed optimization gain.
  3. Concrete diagnostic evidence on widely used metrics. Applying this analysis to PDMS and EPDMS identifies behavioral distinctions that are omitted, thresholded, or attenuated during execution. These results support the broader conclusion that aggregate optimization gains should be interpreted together with the underlying behavioral quantities and the execution process.

We will revise the Introduction and Conclusion to state this hierarchy explicitly and incorporate the new reactive, longer-horizon, and complete-route experiments.

R: Do the increases in jitter, smoothing displacement and jerk reflect real deterioration, or an acceptable trade-off for progress and road compliance?

A: Thank you for raising this point. To test whether these diagnostics relate to real closed-loop driving, we compared Plan-R1 IL and RL within the same nuPlan val14 scene, both executed by nuPlan's LQR tracker in non-reactive traffic. We selected 369 scene pairs in which the two plans have similar acceleration peaks and path lengths, and split them into thirds by the RL − IL change in each diagnostic.

RL − IL CLSRL smootherSimilarRL less smooth
Smoothing displacement+0.034+0.026−0.001
Heading jitter+0.035+0.0240.000

Where RL's plans become less smooth than IL's, RL's closed-loop advantage disappears. These diagnostics therefore relate to closed-loop outcomes, and the same-scene comparison shows whether a loss of smoothness is offset by gains in progress and road compliance.

R: Eq. 3 reuses hkh_k with implicit multiplication, and some numbers in Tables 1 and 3 do not match.

A: Thank you for pointing out these three issues. We will correct the mismatches in Tables 1 and 3 by computing each change from the displayed values rather than from the unrounded means, so that all entries are consistent. We will also report all values to three decimals and label the EPDMS column in each table (Table 1: navtest rescoring of fixed outputs; Table 3: official two-stage navhard score). In Eq. 3, we will replace hkh_k with he,kh_{e,k}, consistent with heh_e in Eq. 2, and add an explicit multiplication sign.

Thank the authors for the detailed response. These clarifications effectively address my questions, and I look forward to seeing these modifications incorporated into the revised manuscript. With my concerns resolved, I am raising my score to 3.

We thank the reviewer for the thoughtful engagement and for raising the score. The questions on the primary finding and its scope helped us state the contribution more precisely. In the revised version, we will distinguish established observations from our new findings in the Introduction and Conclusion, add the reactive, longer-horizon, and complete-route experiments, and correct Tables 1 and 3 and Eq. 3.

Reviewer_3aVcOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies whether gains in multiple driving benchmark scores (NAVSIM v1/v2, nuPlan planning, WorldEngine) remain informative after optimization. It examines omitted behaviors, thresholded scores, and the transformation from requested to executed motion. It also shows that optimization gains can reverse when the execution interface changes.

Critical strengths
  • The matched wrong-way examples give clear evidence of a behavior that PDMS does not distinguish.
  • The study covers multiple planners and evaluation settings, and includes positive cases where optimization improves both scores and behavior.
  • The matched execution comparison controls initial predictions, traffic, scoring, and rendering. The resulting gain reversals are informative.
Critical weaknesses

None

Questions for authors
  • For smoothing and retiming, are the absolute PDMS changes also small per scene, or do positive and negative changes cancel? This affects the evidence for weak score sensitivity.

  • How much do smoothing and retiming change the executed motion? This would clarify whether tracking removes the edits or the score remains insensitive to the resulting motion changes.

3 comments

Thank you for these questions about the smoothing and retiming experiments. We examined per-scene absolute changes and measured the edits before and after tracking to distinguish the possibilities you raised.

R: For smoothing and retiming, small mean PDMS changes could hide large per-scene changes that cancel out, which affects the evidence for weak score sensitivity.

A: Thank you for this question. There is some cancellation, but the per-scene changes are themselves small:

InterventionScenesMean changeMean abs. changeMedian abs. changeAbs. change > 0.05
Qwen-Drive RL retiming (navtest)12,146−0.00020.00240.000635
DiffusionDrive v2 smoothing (navhard)450−0.00290.02180.000518
DiffusionDrive v2 smoothing (WorldEngine)288+0.00670.02340.000412

Half of the scenes change by less than 0.001. The few large changes occur when a thresholded component (collision, drivable area or TTC) flips, which is the thresholding effect discussed in Section 3.

R: Measuring how much smoothing and retiming change the executed motion would show whether tracking removes the edits or the score is insensitive to the remaining change.

A: Tracking attenuates the edits but does not remove them, and the score barely responds to the change that reaches execution:

InterventionRequested editExecuted editExecuted / requested (median)Median abs. score change
Qwen-Drive RL retiming (navtest)0.19 m0.09 m0.550.0006
DiffusionDrive v2 smoothing (navhard)0.42 m0.19 m0.460.0005
DiffusionDrive v2 smoothing (WorldEngine)0.41 m0.19 m0.470.0004

The executed motion changes by about half of the requested edit, so the behavior after execution does change, and the score does not register it.

Thank you for the additional analyses. The per-scene absolute PDMS changes and the requested-versus-executed comparisons substantially address my questions. These results strengthen the evidence for limited score sensitivity to the motion changes studied. I will maintain my rating.

We thank the reviewer for the positive assessment. We will add the per-scene absolute PDMS changes and the requested-versus-executed comparison to the revised version, as they directly strengthen the evidence on score sensitivity.

Reviewer_7Z8LOpenReview ↗
Rating
3
Confidence
4(+1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the question of what improvements in driving capabilities can be demonstrated by score increases when autonomous driving benchmark scores are used as training rewards or search targets. The paper claims that the scoring process discards some important behavioral differences through behavior omission, thresholding, and trajectory execution transformation. Furthermore, even with identical scene, traffic, and scoring conditions, changing only the execution interface can cause the optimization gains of multiple models to turn from positive to negative. Therefore, the interpretation of score increases must simultaneously define "what was measured" and "in what execution and feedback process it holds true."

Critical strengths

The authors incorporated the execution interface into the analysis of benefit effectiveness and used a matched closed-loop comparison to demonstrate that it can change the sign of the optimization benefit. Table 4 compares the two execution interfaces under the same scenario, traffic, scoring, and inference settings, with the initial predictions at each checkpoint being consistent. The RL benefit of Qwen-Drive changes from +0.136 under LQR to -0.388 under playback; ReCogDrive also shows a reversal at both scales, supported by intervals based on driving log bootstrap. This result implies that the claim in existing literature that "RL closed-loop is superior to IL" must specify the execution interface; otherwise, they are not comparable.

Critical weaknesses

The paper measures that the waypoints output by the planner become jittery, but this alone does not indicate a deterioration in driving performance. Jitter, smoothing displacement, and plan-adherence are all calculated on the eight waypoints output by the planner, which are merely intermediate results passed to the controller and do not represent the actual vehicle motion. The paper's own data also shows that the motion changes at the execution layer are minimal: the peak acceleration of ReCogDrive-2B increased from 0.88 to 1.20, while comfort remained consistently high. The paper needs to explain why the geometric changes in waypoints should be considered changes in desired driving behavior.

More importantly, the first plan with a fixed RL output, executed using playback (point-by-point execution) and LQR respectively, showed a PDMS difference of only 0.005. This means that even if the jittery waypoints were executed directly without attenuation, it did not cause a decrease in the score. The benefit reversal in Section 4 stems from the feedback loop of replanning, while Section 3 discusses how the geometric changes in waypoints were not captured by the metrics. These are two separate issues, but the paper strings them together into a unified story without explaining the connection between them.

Questions for authors
  1. Please verify the connection between Sections 3 and 4 using experiments. In particular, explain why the PDMS difference is only -0.005 after freezing the first plan; is this consistent with the explanation of the "execution decay blind zone"?

  2. In a real-world system with a controller, planned waypoints need to pass through the controller to be converted into vehicle actions and are not directly executed. So, why should the waypoint geometry at the request layer also be considered part of the desired driving behavior?

  3. Tables 13/14 show a large number of forward-backward foldbacks after RL, and a single foldback can contribute nearly π of jitter. Is the increase in jitter mainly caused by a few extreme events? Does Figure 2 show the common situation or an occasional occurrence?

  4. In Table 11, the ReCogDrive-2B's IL has a DAC of only 0.66 under LQR, but reaches 0.86 under playback; in four of the five model series, the IL baseline scores are higher under playback. This indicates that, with a 4s closed-loop time and a replanning frequency of 0.5s, LQR tracking itself can cause the conservative plan to deviate from the road surface. Therefore, how much of the improvement in RL under LQR actually compensates for the lag in LQR?

1 comment

Thank you for your thoughtful questions about the connection between the two sections, request-level geometry, and tracking lag. They helped us identify and test the relationships that needed clearer explanation.

R: Freezing the first RL plan changes PDMS by only −0.005 between playback and LQR, so the reversal in Section 4 seems to come from replanning, and its connection to Section 3 is unclear.

A: Thank you for this question; we tested the connection directly. The −0.005 contrast is consistent with Section 3: with the first plan frozen, playback executes the requested +0.42 m/s speed change over the first 0.5 s, whereas LQR executes −0.18 m/s, yet the score barely changes. The reversal then builds up through replanning, since each executed half second becomes the start of the next request. For these analyses we reran the Table 4 setting; the numbers differ slightly from Table 4 (e.g., playback gain −0.380 vs −0.388), but the conclusions are the same. On WorldEngine, the mean Qwen-Drive RL speed rises from 5.5 to 14.9 m/s under playback but falls to 2.9 m/s under LQR, and 166 vs 33 RL rollouts have a collision or road departure by 4 s. Smoothing the speed profile of every request barely changes the interface interaction (+0.517 → +0.485), so request-level oscillation is not what drives the reversal. The two sections are connected through the execution step: Section 3 shows that tracking attenuates request changes by about half, and on WorldEngine, when execution attenuates requests less, the RL gain shrinks (stock LQR +0.136, playback −0.380).

R: In a real system, planned waypoints pass through a controller before being executed, so it is unclear why request-level waypoint geometry should count as driving behavior.

A: We agree that larger request jitter or smoothing displacement does not by itself amount to worse driving, since the vehicle executes the controller's output. We view these quantities as an additional dimension of driving behavior that complements closed-loop evaluation. To test whether this dimension relates to closed-loop outcomes, we compared Plan-R1 IL and RL within the same nuPlan val14 scene under nuPlan's LQR tracker in non-reactive traffic, kept 369 scene pairs with similar planned acceleration peaks and path lengths, and split them into thirds by the RL − IL change in each diagnostic:

RL − IL closed-loop scoreRL smootherSimilarRL less smooth
Smoothing displacement+0.034+0.026−0.001
Heading jitter+0.035+0.0240.000

Where RL's plans become less smooth than IL's, RL's closed-loop advantage disappears. These diagnostics therefore carry information related to closed-loop outcomes, while the closed-loop score remains the primary measure.

R: Tables 13/14 show many forward–backward foldbacks after RL, each contributing nearly π of jitter, so the jitter increase may come from a few extreme events, and Figure 2 may not be a typical case.

A: Thank you for this question. The increase is widespread: on all 12,146 navtest scenes, jitter increases after optimization in 96–97% of scenes for ReCogDrive-2B, ReCogDrive-8B and DiffusionDrive v1→v2. Foldbacks also become common rather than isolated (from 0.5–1.4% to 10–23% of scenes across the three pairs), and the increase remains after removing foldback scenes (ReCogDrive-2B: mean +0.22, median +0.12). Figure 2 shows a high-jitter case, at about the 91st percentile of the 288 WorldEngine scenes, while v2 is more jittery than v1 in 97.6% of these scenes.

R: In Table 11, IL scores higher under playback than under LQR (e.g., ReCogDrive-2B DAC 0.86 vs 0.66), suggesting that LQR tracking lag itself pushes conservative plans off the road, so part of RL's gain under LQR may compensate for this lag.

A: Thank you for this question. We tested it by keeping the LQR unchanged and shortening only the vehicle's actuator response time from 0.2 s to 0.02 s on all 288 WorldEngine scenes. With the faster-responding vehicle, the Qwen-Drive RL gain changes from +0.136 to +0.049, while under playback RL remains below IL (−0.380). The size of the RL gain therefore depends on how quickly the vehicle executes the request, which is one more execution condition under which the gain changes.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.