acceptodds
Under review as a conference paper at ICLR 2027

AttendingScorer: An End-to-End Driving Evaluator that Read Out Trajectory Evaluation Score from latent of Vision-Language-Model

Abstract

Driving vision-language models are now capable of acting as effective evalua- tors between competing trajectories, and providing accurate explanations for their choice. However, they still struggle to provide accurate quantitative scores to these on a non-comparative scale. Existing learned evaluators address this gap by either relying on predefined heuristic rules, or through techniques that involve finetuning the existing comparison-based evaluator to also provide a numerical score. The former approach struggles with unforeseen, long tail events, while the latter involves repeated modification of model weights which can have uninten- tional knock on effects on existing evaluator capability. We propose TrajSteering, a learned approach that aims to read out the trajectory score from the evaluator, AttendingScorer, without touching its weights. It trains only a small set of query embeddings, including a designated score query that aims to get a score through a sum of the context values, weighted by attention, as dictated by the other query embeddings. Our key architectural observation is that attention weights are thus analagous to classical filter taps, with the key distinction being that with filter taps the weights can take on any values, attention weights are restricted to what the frozen model can actually realize. Thus they can be approached with similar prin- ciples. TrajSteering combines two objectives. An automatic pseudo-score loss in- tends to train a scalar head to extract a numeric score from the score-query output, to match targets generated offline by a rule-gated teacher. A deflection-criterion, conditioned on the same teacher, identifies which query embeddings best isolate score-relevant signal from irrelevant noise, with the aim of selecting a readout ca- pable of generalizing among the ones that fit. The backbone remains unchanged; the teacher is used only for one-time offline supervision. Using extensive experi- ments, AttendingScorer perform at a high level of evaluation quality.

Primary area:
applications to robotics, autonomy, planning
Keywords:
VLM, Auto-Driving Evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 2 reviews, above 9% of the others.
Reviewer_E5F3OpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how to produce scalar scores for candidate driving trajectories that better align with human judgments while keeping the vision-language model frozen. It introduces the AttendingScorer framework and its concrete implementation, TrajSteering. The method appends learnable auxiliary queries and a score query to the input, uses the frozen model’s attention to aggregate scene and trajectory information, and predicts a score through a lightweight head. Training uses only automatically generated pseudo-scores from a rule-gated teacher, alongside a teacher-conditioned deflection objective that encourages the attention readout to emphasize score-relevant signals. With comparable pseudo-score fitting performance, the method improves pairwise agreement with human preferences on WOD-E2E from 66.8% for a plain query readout to 70.3%, approaching the 71.5% achieved by LoRA adaptation of the same backbone. Its main contribution is demonstrating the feasibility of lightweight trajectory scoring with a frozen VLM and validating the auxiliary readout objective through targeted controlled experiments.

Critical strengths

The authors propose TrajSteering, which appends trainable queries to a frozen model, aggregates scene and trajectory information through its existing attention mechanism, and predicts scores with a lightweight head. Training only the queries and scoring head achieves performance close to LoRA adaptation, suggesting a lightweight way to reuse existing VLMs for trajectory evaluation.

The comparison with the plain query readout provides a well-controlled test of the auxiliary objective. TrajSteering and the plain query baseline share the same frozen backbone and achieve nearly identical fits to the teacher’s pseudo-scores. Nevertheless, adding the auxiliary objective improves pairwise agreement with human preferences on WOD-E2E from 66.8% to 70.3%, a gain of 3.5 percentage points. This comparison supports an additional generalization benefit beyond improved pseudo-label fitting.

Critical weaknesses
  1. The methodological novelty is relatively limited. The core architecture—appending learnable queries to a frozen VLM and predicting a scalar from the resulting hidden state—closely follows existing query-based readout and prompt-tuning approaches. The main incremental contribution therefore lies in the teacher-conditioned deflection objective. Although the paper provides useful controls against a plain readout and shuffled conditioning targets, it does not establish whether this objective offers a distinctive advantage over simpler supervised representation regularizers under the same architecture and training budget. The filtering interpretation is interesting, but its added algorithmic value remains only partially established.

  2. The practical accuracy–efficiency trade-off is insufficiently quantified. TrajSteering still requires task-specific training and achieves lower human-preference agreement than the same-backbone LoRA scorer, at 70.3% versus 71.5%. This gap would be acceptable if accompanied by a clearly demonstrated reduction in adaptation cost or deployment overhead. However, although the paper states that it compares trainable parameters, GPU-hours, peak training memory, and inference time, the resulting gain is not impressive. Full-backbone SFT is also mentioned in the setup but omitted from the main comparison. Consequently, the evidence does not yet establish when this approach is preferable to standard adaptation methods.

  3. The experimental coverage is narrow relative to the generality of the proposed readout principle. The main comparisons use a single Qwen2.5-VL-7B checkpoint, while external human-alignment evaluation is concentrated on WOD-E2E, with three candidate trajectories per scene. Training on NAVSIM/Pseudo-Simulation and testing on WOD-E2E provides meaningful cross-dataset evidence, but does not establish whether the gains persist across backbone families, model sizes, or broader candidate distributions.

Questions for authors

Please see the weakness section.

5 comments
AuthorsOpenReview ↗

Response to Reviewer E5F3 (1/4): 1.1, 1.2

Response to Reviewer E5F3 — Part 1/4

We deeply appreciate your careful reading and comments towards our work. We clarify the concerns you addressed below.

Weakness 1: Methodological novelty and the added value of TCD

1.1 Contributions beyond query-based readout

Recent work underscores the pressing need for context-aware driving evaluation. DriveJudge [1] combines VLM reasoning with deterministic rule invocation, so its score still depends on explicit rule execution. DriveCritic [2] learns pairwise preferences through SFT and RL, rather than producing a standalone scalar score with a frozen backbone. These approaches leave open how to obtain human-aligned, end-to-end scalar scores without backbone updates or inference-time rule execution. TrajSteering addresses this gap through a teacher-conditioned readout criterion, whose added value is isolated by the matched-fit and auxiliary-objective controls below.

1. Existing evidence. Sections 3–4 distinguish the inherited trailing-query architecture from our contribution: selecting a shared readout within the frozen model’s realizable attention family. The score loss supervises the output, while TCD selects the internal context-attention readout. Table 3 tests whether the location of this criterion matters.

Criterion locationPseudo-validation RMSE ↓WOD pairwise accuracy (%) ↑WOD NDCG@3 ↑
No criterion: plain query readout0.09666.80.89
Full score-query state0.09567.90.90
Context-attention branch: Ours0.09570.30.91

The contribution is the criterion and its placement within the frozen attention interface; at matched teacher fit, the context-attention formulation outperforms applying the same criterion to the full state.

2. Additional experiment. To distinguish this contribution from the query-based readout and prompt-tuning approaches it builds on, we compare the following methods with identical supervision, input representations, training budgets, and validation-only model selection.

ReadoutPseudo-validation RMSE ↓WOD pairwise accuracy (%) ↑WOD MAE ↓Trainable parameters
Prompt tuning (prepended soft prompts + scalar head)0.10165.42.36(K+1)d + d + 1
Prefix tuning (per-layer prefixes + scalar head)0.09866.12.330.23M
Plain query readout0.09666.82.31(K+1)d + d + 1
Query readout + full-state TCD0.09567.92.26(K+1)d + d + 1
Query readout + context-attention TCD0.09570.32.10(K+1)d + d + 1

Prompt tuning, prefix tuning, and the plain trailing-query readout perform comparably, so the inherited parameter-efficient design alone does not account for the result. The gain comes from applying TCD to the context-attention branch, at no additional parameter cost.

Reference

[1] DriveJudge: Rethinking Autonomous Driving Evaluation with Vision-Language Models. arXiv preprint, 2026.

[2] DriveCritic: Towards Context-Aware, Human-Aligned Evaluation for Autonomous Driving with Vision-Language Models. ICRA, 2026.

1.2 Advantage over simpler supervised representation regularizers

1. Existing evidence. Table 3 and Appendix F.6 compare readouts under the same architecture and matched training protocol. In the permutation control, the prediction loss retains the correct labels; only the labels used by TCD are shuffled.

ObjectivePseudo-validation RMSE ↓Pseudo-validation SRCC ↑WOD pairwise accuracy (%) ↑
Prediction loss only0.0960.88866.8
Prediction loss + shuffled-label TCD0.0970.88566.3
Prediction loss + TCD0.0950.89170.3

The gain depends on correctly conditioned supervision rather than improved teacher fit.

2. Additional experiment. We replace only the auxiliary objective while keeping the architecture, training steps, seeds, and hyperparameter-search budget fixed. Signal-only alignment uses normalized latents to prevent gains from trivial scale inflation; the auxiliary regression head adds only a linear layer.

Auxiliary objectivePseudo-validation RMSE ↓WOD pairwise accuracy (%) ↑WOD MAE ↓Training GPU-hours
None0.09666.82.3141.6
Linear prediction of teacher scores from the context latent0.09467.12.2841.8
Normalized signal alignment without residual whitening0.09568.42.2243.4
Target-independent whitening/decorrelation0.09866.42.3342.9
TCD0.09570.32.1043.9

TCD outperforms auxiliary regression, signal alignment, and target-independent whitening at comparable training cost. Combining teacher-conditioned signal alignment with residual whitening yields the strongest external alignment.

AuthorsOpenReview ↗

Response to Reviewer E5F3 (2/4): 1.3, 2.1, 2.2

Response to Reviewer E5F3 — Part 2/4

Weakness 1: Methodological novelty and the added value of TCD

1.3 Evidence for additional algorithmic value

1. Existing evidence. Appendix D.2 registers a falsifiable prediction for the filtering criterion: at matched teacher fit, it should raise held-out deflection together with held-out scoring, most visibly where the teacher is wrong. Tables 3 and 4 test this prediction.

Evidence at matched teacher fitPlain readoutTrajSteeringSource
Held-out ρTCD\rho_{\text{TCD}} ↑6.48.7Table 3
WOD pairwise accuracy (%) ↑66.870.3Table 3
Teacher-blind-spot MAE ↓3.412.79Table 4
TranscendRate (%) ↑52.460.7Table 4

The registered prediction holds: deflection and external alignment rise together, and the gain concentrates on teacher blind spots.

Table 5 tests whether the learned attention allocation affects predictions. Top-attended and random suppression remove equal attention mass.

Intervention on the trained scorerWOD pairwise accuracy (%) ↑Change (percentage points)Held-out ρTCD\rho_{\text{TCD}} ↑
None70.30.08.7
Uniform attention60.9−9.41.2
Top-attended suppression61.8−8.52.6
Random suppression68.9−1.47.9
Within-modality attention shuffling65.1−5.24.8

The learned attention allocation causally contributes to scoring.

2. Additional experiment. We apply the same interventions to Plain, the strongest simple regularizer, and TCD, and test sensitivity to meaning-preserving prompt and rendering changes, which instantiate the nuisance modeled in Section 4.3 (formatting effects and irrelevant semantics). Perturbations are checked to preserve trajectory quality; score drift is the absolute change from each method’s unperturbed score.

MethodClean pairwise accuracy (%) ↑Top-suppression ΔAcc.Random-suppression ΔAcc.Perturbed pairwise accuracy (%) ↑Score drift ↓Rank-flip rate (%) ↓
Plain query readout66.8−4.6−2.163.50.10310.4
Normalized signal alignment68.4−6.0−1.966.60.0888.3
TCD70.3−8.5−1.469.00.0695.6

TCD learns a more selective attention allocation than either alternative, and this selectivity coincides with the highest robustness to meaning-preserving perturbations, which is the nuisance suppression the filtering view predicts.

Weakness 2: Practical accuracy–efficiency trade-off

2.1 When is TrajSteering preferable to standard adaptation?

1. Existing evidence. Table 2 reports overall accuracy, and Table 4 identifies regimes where TrajSteering compares favorably with LoRA.

EvaluationPlain readoutLoRAOurs
Overall pairwise accuracy (%) ↑66.871.570.3
Over-penalty MAE ↓2.882.512.42
Teacher-blind-spot MAE ↓3.413.022.79

TrajSteering achieves the lowest error in the teacher–human disagreement regimes.

2. Additional experiment. We compare all methods under the same GPU-hour cap.

Method at the same GPU-hour capOverall pairwise accuracy (%) ↑Over-penalty MAE ↓Blind-spot MAE ↓
Plain query readout66.82.883.41
LoRA69.82.613.12
Full-backbone SFT68.42.693.31
TrajSteering70.32.422.79

Under a matched compute budget, TrajSteering is the most accurate method overall and in every disagreement regime.

2.2 Quantifying the accuracy–efficiency trade-off

1. Existing evidence. Table 2 compares trajectory-scoring accuracy across adaptation methods. The matched-budget and training-cost evaluations below establish TrajSteering’s adaptation efficiency.

2. Additional experiment. We report accuracy-versus-GPU-hour curves using the same predeclared budget checkpoints for each method. Hyperparameters and checkpoint selection use only pseudo-validation data; external human ratings remain a locked test.

Method and budget checkpointGPU-hoursPairwise accuracy (%) ↑
Plain — low budget1165.3
Plain — medium budget2266.5
Plain — high budget4466.8
LoRA — low budget1163.1
LoRA — medium budget2268.2
LoRA — high budget4469.8
Full SFT — low budget1162.4
Full SFT — medium budget2265.1
Full SFT — high budget4468.4
Ours — low budget1168.1
Ours — medium budget2269.0
Ours — high budget4470.3

TrajSteering is the most accurate method at every budget checkpoint, with the largest margin at the lowest budget.

AuthorsOpenReview ↗

Response to Reviewer E5F3 (3/4): 2.3, 2.4, 3.1

Response to Reviewer E5F3 — Part 3/4

Weakness 2: Practical accuracy–efficiency trade-off

2.3 Reduction in adaptation cost or deployment overhead

1. Existing evidence. Sections 4.1–4.3 and Appendix D.1 explain the structural cost reductions.

ComponentExisting evidenceLocation
Backbone adaptationBackbone remains frozen; only queries and the scalar head are trainedSections 4.1–4.2
Trainable parameter countApproximately (K+1)d + d + 1 for a linear headAppendix D.1
Teacher calls during optimizationNone; labels are generated offline and storedAppendix D.1
TCD computation at inferenceRemovedSection 4.1; Appendix D.1
Human labels for trainingNone; pseudo scores are generated automatically onceSection 5.1; Table 2
Adding an evaluation dimensionTrain only a new query bank and head on the same frozen backboneSection 1; Appendix D.1

The task-specific training that TrajSteering requires is confined to the appended queries and head, uses no human labels, and leaves the shared backbone untouched.

2. Additional experiment. We measure adaptation and inference separately on identical hardware, precision, inputs, and batch settings. Training measurements include TCD computation; offline labeling costs are listed separately. Inference uses merged LoRA.

MethodTrainable parametersTraining GPU-hours ↓Training peak GB ↓Batch-1 latency (ms) ↓
Plain query readout(K+1)d + d + 141.631.4406
LoRA40.4M121.758.7413
Full-backbone SFT8.3B37276.2 ×8413
TrajSteering(K+1)d + d + 143.932.3409

TrajSteering substantially reduces training compute and memory relative to both weight-modifying baselines, with no added inference latency.

Because the appended queries leave the original context states bit-identical (Appendix F.1), TrajSteering leaves the backbone parameters unchanged and retains access to its original inference path, whereas merged LoRA and full SFT modify the backbone weights.

2.4 Missing full-backbone SFT comparison

1. Existing evidence. Section 5.1 mentions full-backbone SFT. We report its performance at matched compute and at pseudo-validation convergence below.

2. Additional experiment. We add full-backbone SFT with the same initialization, training data, rendering, and scalar head, first at matched GPU-hours and then to pseudo-validation convergence. This separates performance under a resource cap from performance after fuller optimization.

MethodTraining conditionGPU-hours ↓Pairwise accuracy (%) ↑MAE ↓Peak training GB ↓
Full SFTMatched GPU-hours4468.42.2176.2 ×8
TrajSteeringMatched GPU-hours43.970.32.1032.3
Full SFTPseudo-validation convergence37269.92.1576.2 ×8

Full-backbone SFT does not surpass TrajSteering at the matched budget, nor after training to convergence with far greater compute.

Weakness 3: Experimental coverage and generalization

3.1 Backbone families and model sizes

1. Existing evidence. The original comparison (Section 5.1; Table 3) fixes the backbone to isolate the readout objective. We extend this comparison across model sizes and backbone families below.

2. Additional experiment. We additionally evaluate Qwen2.5-VL-3B (same family, smaller size) and InternVL2.5-8B (different family, similar size). Each backbone receives a separately trained readout, with matched data and training protocols across its methods.

Backbone settingMethodPseudo-validation RMSE ↓WOD pairwise accuracy (%) ↑MAE ↓Gain over Plain (pp)
Qwen2.5-VL-7BPlain0.09666.82.31—
Qwen2.5-VL-7BTCD0.09570.32.10+3.5
Qwen2.5-VL-3B (same family, smaller)Plain0.10463.72.46—
Qwen2.5-VL-3B (same family, smaller)TCD0.10366.62.30+2.9
InternVL2.5-8B (different family, similar size)Plain0.09965.82.37—
InternVL2.5-8B (different family, similar size)TCD0.09869.12.17+3.3

TCD improves over Plain on all evaluated backbones at matched teacher fit.

Reviewer_XEnAOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies whether a frozen vision-language model can provide useful scalar evaluations of driving trajectories through a learned query interface. The proposed method, TrajSteering, appends trainable queries to a frozen backbone and predicts a score from a designated query state. Training combines regression to automatically generated rule-gated teacher scores with a teacher-conditioned deflection (TCD) objective applied to the context-attention contribution. The central claim is that this objective selects readouts that generalize better than ordinary pseudo-label regression. On WOD-E2E, the paper reports an improvement in pairwise accuracy from 66.8% for plain query readout to 70.3% for TrajSteering, compared with 71.5% for a same-backbone LoRA scorer.

Critical strengths

The problem is practically relevant. Obtaining useful trajectory scores while preserving backbone parameters could make evaluator adaptation easier to deploy and maintain.

The experimental design includes several appropriate controls. In particular, comparing against a plain query readout at approximately matched pseudo-label fit directly addresses the incremental contribution of TCD. The permuted-target and full-state controls also help distinguish the proposed criterion from a generic auxiliary objective.

The paper appropriately acknowledges that classical filtering guarantees do not automatically transfer to the learned attention interface, and that exceeding the teacher is an empirical possibility rather than a theoretical guarantee.

Critical weaknesses
  1. The teacher-blind-spot results require a more cautious interpretation. TCD defines relevant variation through teacher labels, although the motivating failure cases involve information those labels omit. Table 4 provides encouraging evidence of improvement over the teacher, but subgroup sizes and uncertainty are missing. Simple score-shrinkage controls would also help determine whether lower disagreement-subset MAE reflects semantic discrimination or partial regression toward average scores. This concern primarily limits the stronger weak-to-strong interpretation, rather than the basic frozen-readout contribution.
  2. The evidence does not establish the stronger representation-level claim. The introduction motivates the method through a frozen probe that supposedly demonstrates an existing human-aligned scoring signal. However, Appendix C describes the intended protocol without providing the quantitative probe results needed to establish that premise. Successful training of query embeddings demonstrates useful adaptation with a frozen backbone; it does not, by itself, establish recovery of a previously accessible scoring capability or proximity to a measured representation ceiling. The conclusions should distinguish these claims.
Questions for authors
  1. How sensitive are the results to the ridge and latent norms, and where are the quantitative frozen-probe results underlying the introduction’s premise?
  2. Does the implementation use raw or context-renormalized attention weights? How is context attention mass handled?
6 comments
AuthorsOpenReview ↗

Response to Reviewer XEnA (1/5): Weakness 1 (1.1)

Response to Reviewer XEnA — Part 1/5

We deeply appreciate your careful reading and comments towards our work. We clarify the concerns you addressed below.

Weakness 1: Interpretation of the teacher-blind-spot results

1.1 How a teacher-conditioned criterion improves where teacher labels are wrong

1. Existing evidence. Sections 4.2–4.3 and Appendix H state the mechanism. Endpoint supervision fixes what to predict but not how to read: teacher labels underdetermine which frozen features reproduce them, and TCD biases this choice toward score-aligned structure already represented by the backbone. The teacher therefore selects the readout direction, while the information being read comes from the frozen representation. Table 3 tests this at matched teacher fit.

VariantPseudo-validation RMSE ↓Pseudo-validation SRCC ↑WOD pairwise accuracy (%) ↑
Plain query readout0.0960.88866.8
TCD with permuted pseudo scores0.0970.88566.3
TCD on full score-query state0.0950.88967.9
TrajSteering0.0950.89170.3

Readouts with indistinguishable teacher fit differ in human alignment, so teacher fit does not determine what the readout captures.

2. Additional experiment. We test directly whether TrajSteering reads trajectory-quality information that the teacher scores do not carry. Teacher-tied pairs are within-segment candidate pairs whose teacher scores differ by less than 0.5 on the 0–10 scale while their human ratings differ. The teacher cannot order these pairs, so accuracy well above chance reflects information its scores do not carry for them. The trajectory-only scorer of Table 2, trained on the same pseudo scores without camera or language input, shows what fitting the teacher alone provides. Intervals combine five seeds with paired bootstrap over complete driving segments.

ScorerTeacher-tied pair accuracy (%) ↑ (286 pairs, 231 segments)Difference from Plain (pp) [95% interval]
Rule-gated teacher50.7–
Trajectory-only MLP scorer (same pseudo scores)52.4–
Plain query readout55.9–
TCD with permuted pseudo scores55.2−0.7 [−3.9, +2.4]
TCD on full score-query state57.1+1.2 [−2.3, +4.8]
LoRA59.4+3.5 [−0.6, +7.7]
TrajSteering63.2+7.3 [+3.4, +11.5]

We also evaluate on the 11 official WOD-E2E scenario clusters, which tag every segment with one scenario type independently of any score (Xu et al., 2025).

Scenario clusterSegmentsTeacher acc. (%) ↑Plain acc. (%) ↑TrajSteering acc. (%) ↑Teacher MAE ↓Plain MAE ↓TrajSteering MAE ↓
Construction3457.363.168.83.362.662.28
Intersection10864.669.071.22.462.142.02
Pedestrians6161.566.070.02.902.382.13
Cyclists2662.867.369.92.662.262.10
Multi-Lane Maneuvers3166.770.972.42.242.001.93
Single-Lane Maneuvers2766.272.173.32.191.951.88
Cut-ins2463.668.070.62.582.222.06
Foreign Object Debris8258.463.969.63.272.592.23
Special Vehicles2958.964.469.33.192.532.18
Spotlight2260.765.568.92.882.412.17
Others3560.865.768.82.602.122.05
All validation segments47961.966.870.32.782.312.10

The teacher is at chance on pairs it cannot order, and fitting its scores without the frozen representation stays near chance, yet TrajSteering ranks these pairs well above chance and is the only scorer whose improvement over Plain excludes zero, while permuted conditioning yields no gain. Across scenario clusters defined independently of the teacher, TrajSteering improves over both the teacher and Plain in every cluster, most strongly in the construction, debris, and special-vehicle clusters. TCD's teacher conditioning selects how the frozen representation is read; the information that corrects the teacher comes from the representation itself.

AuthorsOpenReview ↗

Response to Reviewer XEnA (2/5): Weakness 1 (1.2–1.3)

Response to Reviewer XEnA — Part 2/5

1.2 Subgroup sizes and uncertainty

1. Existing evidence. Appendix F.7.2 constructs every subgroup automatically from the fixed teacher score, rule outputs, and existing human ratings, with the margin and rule thresholds frozen before WOD-E2E results are inspected. Appendix F.6 specifies seed-level uncertainty and paired bootstrap over complete driving segments. In Table 4, TrajSteering attains the lowest error in both disagreement regimes and the highest TranscendRate.

2. Additional experiment. We report subgroup sizes and 95% intervals that combine five seeds with paired bootstrap over complete driving segments (Appendix F.6). The teacher is deterministic, so its intervals reflect bootstrap only.

WOD-E2E subgroupTrajectories (segments)TeacherPlain readoutLoRATrajSteering
Agreement MAE ↓640 (371)1.58 [1.47, 1.70]1.66 [1.54, 1.79]1.49 [1.38, 1.61]1.61 [1.49, 1.72]
Over-penalty MAE ↓360 (247)3.42 [3.19, 3.66]2.88 [2.66, 3.12]2.51 [2.29, 2.72]2.42 [2.21, 2.65]
Blind-spot MAE ↓200 (152)3.95 [3.61, 4.27]3.41 [3.07, 3.77]3.02 [2.71, 3.36]2.79 [2.49, 3.12]
TranscendRate (%) ↑797 (412)–52.4 [48.1, 56.9]56.1 [51.8, 60.2]60.7 [56.6, 64.9]
Paired difference (95% interval)TrajSteering − TeacherTrajSteering − Plain
Over-penalty MAE ↓−1.00 [−1.24, −0.77]−0.46 [−0.63, −0.30]
Blind-spot MAE ↓−1.16 [−1.47, −0.83]−0.62 [−0.85, −0.40]
TranscendRate (pp) ↑–+8.3 [+4.6, +12.1]

Every subgroup spans many independent driving segments, and TrajSteering's improvements over both the teacher and Plain in the disagreement regimes hold with paired intervals that exclude zero.

1.3 Score-shrinkage controls

1. Existing evidence. Table 2 reports within-scene ranking, which is invariant to any monotone shrinkage of the scores.

EvidencePlain readoutTrajSteeringSource
Kendall τb ↑0.330.40Table 2
NDCG@3 ↑0.890.91Table 2

TrajSteering ranks candidates better than Plain, an improvement that shrinkage toward the mean cannot produce.

2. Additional experiment. We shrink the teacher and Plain scores toward the training-pool mean, y^γ=μ+γ(y^−μ)\hat{y}_\gamma = \mu + \gamma(\hat{y} - \mu), with γ\gamma chosen to minimize overall MAE directly on the WOD-E2E human ratings. This oracle tuning favors the controls, since TrajSteering never sees these ratings. Disagreement-pair accuracy is pairwise accuracy over unequal-rating within-segment pairs whose two trajectories both lie in the disagreement set DdisD_{\text{dis}} (Appendix F.7.2).

ScorerAgreement MAE ↓Over-penalty MAE ↓Blind-spot MAE ↓Disagreement-pair accuracy (%) ↑
Teacher1.583.423.9547.2
Teacher, oracle shrinkage (γ = 0.61)1.972.983.3147.2
Plain readout1.662.883.4158.6
Plain readout, oracle shrinkage (γ = 0.80)1.812.703.1658.6
TrajSteering1.612.422.7964.9

Shrinkage lowers disagreement-regime error only by raising agreement error, leaves ranking unchanged, and remains short of TrajSteering even with oracle tuning. TrajSteering reduces error in every regime relative to Plain and improves ranking on disagreement pairs, so its gain reflects semantic discrimination rather than regression toward the mean.

AuthorsOpenReview ↗

Response to Reviewer XEnA (3/5): Weakness 2

Response to Reviewer XEnA — Part 3/5

Weakness 2: Evidence for the representation-level premise

2.1 Quantitative frozen-probe results

1. Existing evidence. Appendix C fixes the probe protocol and binding falsification conditions before training; condition (a) requires the frozen probe to land clearly above both the rule score and direct prompting. Figure 4 shows that frozen readouts on the same backbone exceed every prompting control.

Scorer on the same frozen backboneWOD pairwise accuracy (%) ↑Effective score levels ↑Prompt sensitivity ↓
Strongest prompting control57.623.70.62
Plain query readout66.82870.11
TrajSteering70.33410.09

Stronger elicitation does not close the gap to frozen readouts; the native textual channel, not the representation, limits prompting.

2. Additional experiment. We report probes following the Appendix C protocol. Linear and two-layer MLP probes read pooled final-layer states of the original context, which are bit-identical with or without appended queries (Appendix F.1). As in Appendix C, probes are trained on the DriveCritic training preference pairs, where rule scores and expert judgment disagree (Song et al., 2026), and are evaluated on WOD-E2E, which remains a locked test (Section 5.1); probes never train, select, or calibrate any scorer. A shuffled-label probe, trained on the same pairs with permuted preferences, checks that the probe pipeline yields no above-chance accuracy without the true labels, and, as Appendix C requires, the protocol is repeated on a second backbone, InternVL2.5-8B. Disagreement pairs are within-segment pairs whose two trajectories both lie in DdisD_{\text{dis}}.

Pairwise accuracy (%) ↑Qwen2.5-VL-7B, all pairsQwen2.5-VL-7B, disagreement pairsInternVL2.5-8B, all pairsInternVL2.5-8B, disagreement pairs
EPDMS rule score59.744.859.744.8
Rule-gated teacher61.947.261.947.2
Direct prompting54.251.352.650.6
Strongest prompting control57.652.955.852.1
Shuffled-label probe50.949.650.449.8
Frozen linear probe73.167.871.766.2
Frozen MLP probe74.669.573.267.9

On both backbones, frozen probes land clearly above the rule score, the teacher, and every prompting control, including on teacher–human disagreement pairs, while the shuffled-label probe remains at chance. The human-aligned scoring signal is present in the frozen representation, and falsification condition (a) of Appendix C is met.

2.2 Recovering an accessible signal versus successful adaptation

1. Existing evidence. Appendix F.1 verifies that the appended queries leave the original context states bit-identical, so the probes, untrained queries, and TrajSteering all read the same states. Table 3 includes fixed random queries trained with only a scalar head. These untrained queries aggregate the frozen states only through the model's own attention, yet exceed every prompting control (Figure 4), so the signal is accessible through the native interface before any query is trained.

2. Additional experiment. We place each scorer relative to the measured ceiling, the best frozen probe in our response to W2.1. The recovered gap is (accuracy − direct prompting) / (ceiling − direct prompting).

ScorerSupervisionAll-pair accuracy (%) ↑Disagreement-pair accuracy (%) ↑Gap recovered, all pairs (%) ↑Gap recovered, disagreement pairs (%) ↑
Direct promptingNone54.251.30.00.0
Fixed random queriesPseudo scores (scalar head only)61.554.435.817.0
Plain query readoutPseudo scores66.858.661.840.1
LoRAPseudo scores71.562.884.863.2
TrajSteeringPseudo scores70.364.978.974.7
Frozen-probe ceilingExpert preference pairs (Appendix C)74.669.5100.0100.0

Each claim rests on its own evidence: training frozen queries is effective (Table 2); the human-aligned signal is present and accessible in the untouched states (our response to W2.1; untrained queries); and TrajSteering, trained without human labels, recovers most of the measured gap to the ceiling, with the largest share of any scorer on teacher–human disagreement pairs.

Comments on the paper

4 comments
AuthorsOpenReview ↗

Response to Reviewer E3ke (1/3): Weakness 1

Response to Reviewer E3ke — Part 1/3

We greatly appreciate your kind review and support to this work.

Weakness 1: Teacher-unexplained variation and generality across teacher errors

1.1 Whether whitening teacher-unexplained variation discards overlooked driving information

1. Existing evidence. Sections 3, 4.1, and 4.3 and Algorithm 2 separate what TCD shapes from what the score is computed on.

ComponentTreatment of variation the teacher does not explainLocation
Score headReads the full score-query state hsh_s; the criterion adds no projection, mask, or filter to the scoring pathSections 3 and 4.3
TCD criterionApplied only to the context-attention latent zEz_E; down-weights, per latent coordinate, variance that the teacher does not explain, without projecting it outAlgorithm 2
InferenceThe criterion is discarded; the score is produced from hsh_s aloneSection 4.1

TCD acts only through the query bank during training: the frozen context states stay unchanged and fully visible to the score query, no projection or mask is applied to hsh_s, and the score head reads hsh_s in full.

2. Additional experiment. We compare objectives that differ in how they treat variation that the teacher does not explain, keeping supervision, architecture, and training budget fixed. The variants are no criterion (Plain), normalized signal alignment without residual whitening (the auxiliary objective in our response to Reviewer E5F3), TCD with its residual whitening weakened by a hundredfold ridge (the ridge sweep in our response to Reviewer XEnA), and TCD. Teacher-tied pairs are within-segment pairs whose teacher scores differ by less than 0.5 on the 0–10 scale while their human ratings differ; accuracy on them measures information that the teacher's scores do not carry.

Treatment of teacher-unexplained variationWOD pairwise accuracy (%) ↑Over-penalty MAE ↓Blind-spot MAE ↓Teacher-tied pair accuracy (%) ↑
No criterion (Plain)66.82.883.4155.9
Normalized signal alignment, residual not whitened68.42.713.0858.3
TCD, residual whitening weakened (εD×100\varepsilon_D \times 100)68.72.693.0558.8
TCD (default)70.32.422.7963.2

Weakening or removing the residual whitening degrades the cases the teacher gets wrong and the pairs it cannot order, so the variation that TCD whitens behaves as nuisance rather than as overlooked driving information. The information that corrects the teacher is retained and read more selectively.

1.2 Generality across types of teacher errors

1. Existing evidence. Table 4 separates teacher errors of opposite direction (Appendix F.7.2): over-penalty, where the teacher scores a trajectory more than the margin δ\delta below its human rating, and blind spots, where every represented rule passes but the teacher scores the trajectory more than δ\delta above its human rating.

WOD-E2E subgroupTeacherPlain readoutOurs
Over-penalty MAE ↓3.422.882.42
Blind-spot MAE ↓3.953.412.79
TranscendRate (%) ↑–52.460.7

TrajSteering improves on the teacher for teacher errors in both directions.

2. Additional experiment. We break the over-penalty subset down by the rule whose repair (setting its rule score to one) raises the teacher score most, computed automatically from the gated public rule outputs that form the teacher target (Appendix F.2), with ties broken by the fixed order of M\mathcal{M}. The blind-spot subset, where every rule passes, is broken down by driving behavior in our response to Q2.

Rule driving the teacher's over-penaltyTypical over-penalized behaviorTrajectoriesTeacher MAE ↓Plain MAE ↓TrajSteering MAE ↓
No collision (NC)Contact flagged for non-hazardous objects214.363.522.86
Drivable-area compliance (DAC)Brief shoulder use to yield or clear an obstacle635.244.143.29
Driving-direction compliance (DDC)Brief opposite-lane use to pass a blockage374.053.312.71
Time to collision (TTC)Accepted close gaps in dense traffic793.022.662.27
Ego progress (EP)Deliberate slowing or stopping for safety662.962.612.24
Lane keeping (LK)Lateral offset to keep clearance582.482.181.93
ComfortWarranted hard braking362.271.971.79
All over-penalty–3603.422.882.42

TrajSteering improves over both the teacher and Plain whichever rule drives the over-penalty, from the multiplicative safety gates to the averaged progress, lane-keeping, and comfort terms, so its gain is not tied to one type of teacher error.

AuthorsOpenReview ↗

Response to Reviewer E3ke (2/3): Weakness 2 (2.1–2.2)

Response to Reviewer E3ke — Part 2/3

Weakness 2: Connecting Equation 2 to the implementation

2.1 Attention mask of the score query

1. Existing evidence. Section 3 appends the KK auxiliary queries and the score query after the original context under the model's standard causal mask, so the appended queries attend to all preceding positions while every original context state stays unchanged (verified bit-identical in Appendix F.1). The score query ese_s is the last appended position.

Positions visible to the score queryUnder the causal mask
Original context C\mathcal{C}Yes
Auxiliary queries e1,…,eKe_1, \dots, e_KYes
Score query itselfYes
Effect on original context statesNone (bit-identical, Appendix F.1)

No special mask is used, so the raw weights on the original context sum to the context mass m(r)≤1m^{(r)} \le 1, and Eq. 2 normalizes them over the context.

2. Additional experiment. We measure on WOD-E2E how the final-layer score-query heads distribute attention across the three groups of positions.

MethodMass on original contextMass on auxiliary queriesMass on score query itself
Plain query readout0.910.060.03
TrajSteering0.890.080.03

Most of the score query's attention falls on the original context, and training with TCD leaves this allocation nearly unchanged.

2.2 Whether the context weights are renormalized

1. Existing evidence. Eq. 2 defines the context branch of the score-query attention: its weights α(r)\alpha^{(r)} are the score-query weights normalized over the original context, so they lie on the simplex. With raw weights a(r)a^{(r)} over all preceding positions and context mass m(r)=∑i∈Cai(r)m^{(r)} = \sum_{i \in \mathcal{C}} a_i^{(r)}, the score-query attention output decomposes exactly as o(r)=m(r)oE(r)+(1−m(r))oQ(r)o^{(r)} = m^{(r)} o_E^{(r)} + (1 - m^{(r)}) o_Q^{(r)}, where oE(r)=∑i∈Cαi(r)vi(r)o_E^{(r)} = \sum_{i \in \mathcal{C}} \alpha_i^{(r)} v_i^{(r)} is the context branch of Eq. 2 and oQ(r)o_Q^{(r)} is the branch over the appended queries. The TCD latent is built from the context branch (Eq. 3).

QuantityDefinitionLocation
Raw score-query weights ai(r)a_i^{(r)}Softmax over all preceding positions (original context, auxiliary queries, score query)Section 3 (causal readout)
Normalized taps αi(r)\alpha_i^{(r)}ai(r)/m(r)∈Δ∣C∣−1a_i^{(r)} / m^{(r)} \in \Delta^{\lvert\mathcal{C}\rvert-1}Eq. 2
Context branch oE(r)o_E^{(r)}∑i∈Cαi(r)vi(r)\sum_{i \in \mathcal{C}} \alpha_i^{(r)} v_i^{(r)}Eq. 2
Exact decompositiono(r)=m(r)oE(r)+(1−m(r))oQ(r)o^{(r)} = m^{(r)} o_E^{(r)} + (1 - m^{(r)}) o_Q^{(r)}, with mixing weight m(r)=∑i∈Cai(r)m^{(r)} = \sum_{i \in \mathcal{C}} a_i^{(r)}Follows from Eq. 2
Context-attention latent zEz_E∑rWO(r)oE(r)\sum_r W_O^{(r)} o_E^{(r)}Eq. 3

The context weights are normalized over the original context exactly as Eq. 2 states, and the context mass enters the score-query output as the exact mixing weight of the context branch.

2. Additional experiment. We verify Eq. 2 on the actual attention branch. On a held-out batch, we recompute each final-layer score-query head's output from the hooked attention weights and values and compare (i) the exact decomposition with the model's actual attention output, (ii) the latent passed to TCD in the training code with the Eq. 3 form, and (iii) the same latent with a raw-weight variant that multiplies the context mass into the latent. Relative error is ∥x−y∥/∥y∥\lVert x - y \rVert / \lVert y \rVert; the verification forward pass runs in fp32 with eager attention so that attention weights are materialized.

Check on final-layer score-query headsMean relative errorMax relative error
(i) m(r)oE(r)+(1−m(r))oQ(r)m^{(r)} o_E^{(r)} + (1 - m^{(r)}) o_Q^{(r)} vs. actual attention output2.1×10−72.1 \times 10^{-7}9.4×10−79.4 \times 10^{-7}
(ii) Latent passed to TCD vs. Eq. 3 form ∑rWO(r)∑i∈Cαi(r)vi(r)\sum_r W_O^{(r)} \sum_{i \in \mathcal{C}} \alpha_i^{(r)} v_i^{(r)}3.6×10−83.6 \times 10^{-8}1.5×10−71.5 \times 10^{-7}
(iii) Latent passed to TCD vs. raw-weight variant ∑rWO(r)∑i∈Cai(r)vi(r)\sum_r W_O^{(r)} \sum_{i \in \mathcal{C}} a_i^{(r)} v_i^{(r)}1.3×10−11.3 \times 10^{-1}4.1×10−14.1 \times 10^{-1}

The exact decomposition reproduces the actual attention output to floating-point precision, and the latent used in training matches Eq. 3 while clearly differing from the raw-weight variant, so the implementation follows Eq. 2 as written.

AuthorsOpenReview ↗

Response to Reviewer E3ke (3/3): Weakness 2 (2.3), Questions 1–2

Response to Reviewer E3ke — Part 3/3

2.3 Whether total context mass is retained, and whether the choice matters

1. Existing evidence. The forward pass retains each head's context mass through the exact decomposition, so the score head reads the full attention output. The TCD latent isolates the context branch, whose normalized taps the Table 5 interventions redistribute and renormalize (Appendix F.7.3).

2. Additional experiment. We compare the three options the reviewer lists. TrajSteering is retrained with a raw-weight latent that multiplies each head's context mass into zEz_E, and both methods are retrained with a context-only mask in the final layer, under which the score query attends only to the original context and m(r)=1m^{(r)} = 1. Plain, with λ=0\lambda = 0, does not depend on the latent definition, so under the raw-weight latent only its held-out ρTCD\rho_{\text{TCD}} is re-evaluated.

Latent definitionMethodWOD pairwise accuracy (%) ↑Blind-spot MAE ↓Held-out ρTCD\rho_{\text{TCD}} ↑Mean context mass
Context-normalized branch (Eq. 2, default)Plain66.83.416.40.91
Context-normalized branch (Eq. 2, default)TrajSteering70.32.798.70.89
Raw weights, mass multiplied into zEz_EPlain66.83.416.20.91
Raw weights, mass multiplied into zEz_ETrajSteering69.82.878.30.88
Context-only final-layer maskPlain66.13.476.01.00
Context-only final-layer maskTrajSteering69.62.888.21.00

TCD's gain over Plain holds whether the context mass is excluded from the latent, multiplied into it, or fixed to one by a context-only mask, and training leaves the context mass nearly unchanged; the gain comes from how attention is distributed over context positions rather than from how much attention the context receives.

Question 1: Attention mask and normalization in Equation 2

1. Existing evidence. Section 3 places the appended queries after the original context under the standard causal mask, and Eqs. 2–3 define the context branch and the TCD latent built from it.

2. Additional experiment. The full answer is given in our response to W2: W2.1 reports the mask and the attention-mass decomposition, W2.2 verifies Eq. 2 on the actual attention branch, and W2.3 reports results with the context mass excluded from the latent, multiplied into it, or fixed to one.

The score query attends to the original context, the auxiliary queries, and itself; the context weights are normalized over the context as Eq. 2 states, the total context-attention mass is retained as the exact mixing weight of the context branch, and the TCD latent is built from the normalized branch.

Question 2: Driving behaviors behind the blind-spot improvement

1. Existing evidence. Appendix F.7.2 defines the blind-spot subset automatically as trajectories for which every represented rule passes while the existing human rating indicates poor quality, and Appendix H characterizes it as the regime in which a rule gate cannot introduce a penalty that no rule expresses. In Table 4, TrajSteering has the lowest blind-spot error.

2. Additional experiment. We break the blind-spot subset down by driving behavior. Each trajectory inherits the behavior family of its segment's official WOD-E2E scenario cluster (Xu et al., 2025), so the assignment is automatic and independent of any score; no new annotation is collected.

Behavior family (WOD-E2E clusters)Scene semantics that no rule encodesTrajectoriesTeacher MAE ↓Plain MAE ↓TrajSteering MAE ↓
Vulnerable road users (Pedestrians, Cyclists)Pedestrian or cyclist intent to cross414.123.522.71
Construction zones (Construction)Closed or flagged lanes and temporary traffic control234.313.712.83
Special vehicles (Special Vehicles)Emergency or service vehicles that require yielding194.263.662.79
Debris and unusual objects (Foreign Object Debris)Debris or animals in or near the driving path383.953.432.74
Right-of-way and negotiation (Intersection, Cut-ins, Multi-Lane, Single-Lane Maneuvers)Priority expectations at unprotected turns, merges, and lane changes583.623.132.83
Other long-tail (Spotlight, Others)Remaining long-tail scene semantics213.843.372.88
All blind spots–2003.953.412.79

The improvement over both the teacher and Plain holds in every behavior family of the blind-spot subset, with the largest gains in construction zones, special vehicles, vulnerable road users, and debris, whose scenes carry semantics that no rule encodes, so the benefit extends across semantic error types rather than being concentrated in a single case type.

Please feel free to ask any questions if there remains, and we are glad to answer. Thank you again!

Thank you for the detailed response and additional experiments. The additional clarifications and results have addressed my main concerns. Overall, I maintain my positive assessment of the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.