- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates how to produce scalar scores for candidate driving trajectories that better align with human judgments while keeping the vision-language model frozen. It introduces the AttendingScorer framework and its concrete implementation, TrajSteering. The method appends learnable auxiliary queries and a score query to the input, uses the frozen model’s attention to aggregate scene and trajectory information, and predicts a score through a lightweight head. Training uses only automatically generated pseudo-scores from a rule-gated teacher, alongside a teacher-conditioned deflection objective that encourages the attention readout to emphasize score-relevant signals. With comparable pseudo-score fitting performance, the method improves pairwise agreement with human preferences on WOD-E2E from 66.8% for a plain query readout to 70.3%, approaching the 71.5% achieved by LoRA adaptation of the same backbone. Its main contribution is demonstrating the feasibility of lightweight trajectory scoring with a frozen VLM and validating the auxiliary readout objective through targeted controlled experiments.
Critical strengths
The authors propose TrajSteering, which appends trainable queries to a frozen model, aggregates scene and trajectory information through its existing attention mechanism, and predicts scores with a lightweight head. Training only the queries and scoring head achieves performance close to LoRA adaptation, suggesting a lightweight way to reuse existing VLMs for trajectory evaluation.
The comparison with the plain query readout provides a well-controlled test of the auxiliary objective. TrajSteering and the plain query baseline share the same frozen backbone and achieve nearly identical fits to the teacher’s pseudo-scores. Nevertheless, adding the auxiliary objective improves pairwise agreement with human preferences on WOD-E2E from 66.8% to 70.3%, a gain of 3.5 percentage points. This comparison supports an additional generalization benefit beyond improved pseudo-label fitting.
Critical weaknesses
-
The methodological novelty is relatively limited. The core architecture—appending learnable queries to a frozen VLM and predicting a scalar from the resulting hidden state—closely follows existing query-based readout and prompt-tuning approaches. The main incremental contribution therefore lies in the teacher-conditioned deflection objective. Although the paper provides useful controls against a plain readout and shuffled conditioning targets, it does not establish whether this objective offers a distinctive advantage over simpler supervised representation regularizers under the same architecture and training budget. The filtering interpretation is interesting, but its added algorithmic value remains only partially established.
-
The practical accuracy–efficiency trade-off is insufficiently quantified. TrajSteering still requires task-specific training and achieves lower human-preference agreement than the same-backbone LoRA scorer, at 70.3% versus 71.5%. This gap would be acceptable if accompanied by a clearly demonstrated reduction in adaptation cost or deployment overhead. However, although the paper states that it compares trainable parameters, GPU-hours, peak training memory, and inference time, the resulting gain is not impressive. Full-backbone SFT is also mentioned in the setup but omitted from the main comparison. Consequently, the evidence does not yet establish when this approach is preferable to standard adaptation methods.
-
The experimental coverage is narrow relative to the generality of the proposed readout principle. The main comparisons use a single Qwen2.5-VL-7B checkpoint, while external human-alignment evaluation is concentrated on WOD-E2E, with three candidate trajectories per scene. Training on NAVSIM/Pseudo-Simulation and testing on WOD-E2E provides meaningful cross-dataset evidence, but does not establish whether the gains persist across backbone families, model sizes, or broader candidate distributions.
Questions for authors
Please see the weakness section.
5 comments
Response to Reviewer E5F3 (2/4): 1.3, 2.1, 2.2
Response to Reviewer E5F3 — Part 2/4
Weakness 1: Methodological novelty and the added value of TCD
1.3 Evidence for additional algorithmic value
1. Existing evidence. Appendix D.2 registers a falsifiable prediction for the filtering criterion: at matched teacher fit, it should raise held-out deflection together with held-out scoring, most visibly where the teacher is wrong. Tables 3 and 4 test this prediction.
| Evidence at matched teacher fit | Plain readout | TrajSteering | Source |
|---|---|---|---|
| Held-out ↑ | 6.4 | 8.7 | Table 3 |
| WOD pairwise accuracy (%) ↑ | 66.8 | 70.3 | Table 3 |
| Teacher-blind-spot MAE ↓ | 3.41 | 2.79 | Table 4 |
| TranscendRate (%) ↑ | 52.4 | 60.7 | Table 4 |
The registered prediction holds: deflection and external alignment rise together, and the gain concentrates on teacher blind spots.
Table 5 tests whether the learned attention allocation affects predictions. Top-attended and random suppression remove equal attention mass.
| Intervention on the trained scorer | WOD pairwise accuracy (%) ↑ | Change (percentage points) | Held-out ↑ |
|---|---|---|---|
| None | 70.3 | 0.0 | 8.7 |
| Uniform attention | 60.9 | −9.4 | 1.2 |
| Top-attended suppression | 61.8 | −8.5 | 2.6 |
| Random suppression | 68.9 | −1.4 | 7.9 |
| Within-modality attention shuffling | 65.1 | −5.2 | 4.8 |
The learned attention allocation causally contributes to scoring.
2. Additional experiment. We apply the same interventions to Plain, the strongest simple regularizer, and TCD, and test sensitivity to meaning-preserving prompt and rendering changes, which instantiate the nuisance modeled in Section 4.3 (formatting effects and irrelevant semantics). Perturbations are checked to preserve trajectory quality; score drift is the absolute change from each method’s unperturbed score.
| Method | Clean pairwise accuracy (%) ↑ | Top-suppression ΔAcc. | Random-suppression ΔAcc. | Perturbed pairwise accuracy (%) ↑ | Score drift ↓ | Rank-flip rate (%) ↓ |
|---|---|---|---|---|---|---|
| Plain query readout | 66.8 | −4.6 | −2.1 | 63.5 | 0.103 | 10.4 |
| Normalized signal alignment | 68.4 | −6.0 | −1.9 | 66.6 | 0.088 | 8.3 |
| TCD | 70.3 | −8.5 | −1.4 | 69.0 | 0.069 | 5.6 |
TCD learns a more selective attention allocation than either alternative, and this selectivity coincides with the highest robustness to meaning-preserving perturbations, which is the nuisance suppression the filtering view predicts.
Weakness 2: Practical accuracy–efficiency trade-off
2.1 When is TrajSteering preferable to standard adaptation?
1. Existing evidence. Table 2 reports overall accuracy, and Table 4 identifies regimes where TrajSteering compares favorably with LoRA.
| Evaluation | Plain readout | LoRA | Ours |
|---|---|---|---|
| Overall pairwise accuracy (%) ↑ | 66.8 | 71.5 | 70.3 |
| Over-penalty MAE ↓ | 2.88 | 2.51 | 2.42 |
| Teacher-blind-spot MAE ↓ | 3.41 | 3.02 | 2.79 |
TrajSteering achieves the lowest error in the teacher–human disagreement regimes.
2. Additional experiment. We compare all methods under the same GPU-hour cap.
| Method at the same GPU-hour cap | Overall pairwise accuracy (%) ↑ | Over-penalty MAE ↓ | Blind-spot MAE ↓ |
|---|---|---|---|
| Plain query readout | 66.8 | 2.88 | 3.41 |
| LoRA | 69.8 | 2.61 | 3.12 |
| Full-backbone SFT | 68.4 | 2.69 | 3.31 |
| TrajSteering | 70.3 | 2.42 | 2.79 |
Under a matched compute budget, TrajSteering is the most accurate method overall and in every disagreement regime.
2.2 Quantifying the accuracy–efficiency trade-off
1. Existing evidence. Table 2 compares trajectory-scoring accuracy across adaptation methods. The matched-budget and training-cost evaluations below establish TrajSteering’s adaptation efficiency.
2. Additional experiment. We report accuracy-versus-GPU-hour curves using the same predeclared budget checkpoints for each method. Hyperparameters and checkpoint selection use only pseudo-validation data; external human ratings remain a locked test.
| Method and budget checkpoint | GPU-hours | Pairwise accuracy (%) ↑ |
|---|---|---|
| Plain — low budget | 11 | 65.3 |
| Plain — medium budget | 22 | 66.5 |
| Plain — high budget | 44 | 66.8 |
| LoRA — low budget | 11 | 63.1 |
| LoRA — medium budget | 22 | 68.2 |
| LoRA — high budget | 44 | 69.8 |
| Full SFT — low budget | 11 | 62.4 |
| Full SFT — medium budget | 22 | 65.1 |
| Full SFT — high budget | 44 | 68.4 |
| Ours — low budget | 11 | 68.1 |
| Ours — medium budget | 22 | 69.0 |
| Ours — high budget | 44 | 70.3 |
TrajSteering is the most accurate method at every budget checkpoint, with the largest margin at the lowest budget.
Response to Reviewer E5F3 (3/4): 2.3, 2.4, 3.1
Response to Reviewer E5F3 — Part 3/4
Weakness 2: Practical accuracy–efficiency trade-off
2.3 Reduction in adaptation cost or deployment overhead
1. Existing evidence. Sections 4.1–4.3 and Appendix D.1 explain the structural cost reductions.
| Component | Existing evidence | Location |
|---|---|---|
| Backbone adaptation | Backbone remains frozen; only queries and the scalar head are trained | Sections 4.1–4.2 |
| Trainable parameter count | Approximately (K+1)d + d + 1 for a linear head | Appendix D.1 |
| Teacher calls during optimization | None; labels are generated offline and stored | Appendix D.1 |
| TCD computation at inference | Removed | Section 4.1; Appendix D.1 |
| Human labels for training | None; pseudo scores are generated automatically once | Section 5.1; Table 2 |
| Adding an evaluation dimension | Train only a new query bank and head on the same frozen backbone | Section 1; Appendix D.1 |
The task-specific training that TrajSteering requires is confined to the appended queries and head, uses no human labels, and leaves the shared backbone untouched.
2. Additional experiment. We measure adaptation and inference separately on identical hardware, precision, inputs, and batch settings. Training measurements include TCD computation; offline labeling costs are listed separately. Inference uses merged LoRA.
| Method | Trainable parameters | Training GPU-hours ↓ | Training peak GB ↓ | Batch-1 latency (ms) ↓ |
|---|---|---|---|---|
| Plain query readout | (K+1)d + d + 1 | 41.6 | 31.4 | 406 |
| LoRA | 40.4M | 121.7 | 58.7 | 413 |
| Full-backbone SFT | 8.3B | 372 | 76.2 ×8 | 413 |
| TrajSteering | (K+1)d + d + 1 | 43.9 | 32.3 | 409 |
TrajSteering substantially reduces training compute and memory relative to both weight-modifying baselines, with no added inference latency.
Because the appended queries leave the original context states bit-identical (Appendix F.1), TrajSteering leaves the backbone parameters unchanged and retains access to its original inference path, whereas merged LoRA and full SFT modify the backbone weights.
2.4 Missing full-backbone SFT comparison
1. Existing evidence. Section 5.1 mentions full-backbone SFT. We report its performance at matched compute and at pseudo-validation convergence below.
2. Additional experiment. We add full-backbone SFT with the same initialization, training data, rendering, and scalar head, first at matched GPU-hours and then to pseudo-validation convergence. This separates performance under a resource cap from performance after fuller optimization.
| Method | Training condition | GPU-hours ↓ | Pairwise accuracy (%) ↑ | MAE ↓ | Peak training GB ↓ |
|---|---|---|---|---|---|
| Full SFT | Matched GPU-hours | 44 | 68.4 | 2.21 | 76.2 ×8 |
| TrajSteering | Matched GPU-hours | 43.9 | 70.3 | 2.10 | 32.3 |
| Full SFT | Pseudo-validation convergence | 372 | 69.9 | 2.15 | 76.2 ×8 |
Full-backbone SFT does not surpass TrajSteering at the matched budget, nor after training to convergence with far greater compute.
Weakness 3: Experimental coverage and generalization
3.1 Backbone families and model sizes
1. Existing evidence. The original comparison (Section 5.1; Table 3) fixes the backbone to isolate the readout objective. We extend this comparison across model sizes and backbone families below.
2. Additional experiment. We additionally evaluate Qwen2.5-VL-3B (same family, smaller size) and InternVL2.5-8B (different family, similar size). Each backbone receives a separately trained readout, with matched data and training protocols across its methods.
| Backbone setting | Method | Pseudo-validation RMSE ↓ | WOD pairwise accuracy (%) ↑ | MAE ↓ | Gain over Plain (pp) |
|---|---|---|---|---|---|
| Qwen2.5-VL-7B | Plain | 0.096 | 66.8 | 2.31 | — |
| Qwen2.5-VL-7B | TCD | 0.095 | 70.3 | 2.10 | +3.5 |
| Qwen2.5-VL-3B (same family, smaller) | Plain | 0.104 | 63.7 | 2.46 | — |
| Qwen2.5-VL-3B (same family, smaller) | TCD | 0.103 | 66.6 | 2.30 | +2.9 |
| InternVL2.5-8B (different family, similar size) | Plain | 0.099 | 65.8 | 2.37 | — |
| InternVL2.5-8B (different family, similar size) | TCD | 0.098 | 69.1 | 2.17 | +3.3 |
TCD improves over Plain on all evaluated backbones at matched teacher fit.
Response to Reviewer E5F3 (1/4): 1.1, 1.2
Response to Reviewer E5F3 — Part 1/4
We deeply appreciate your careful reading and comments towards our work. We clarify the concerns you addressed below.
Weakness 1: Methodological novelty and the added value of TCD
1.1 Contributions beyond query-based readout
Recent work underscores the pressing need for context-aware driving evaluation. DriveJudge [1] combines VLM reasoning with deterministic rule invocation, so its score still depends on explicit rule execution. DriveCritic [2] learns pairwise preferences through SFT and RL, rather than producing a standalone scalar score with a frozen backbone. These approaches leave open how to obtain human-aligned, end-to-end scalar scores without backbone updates or inference-time rule execution. TrajSteering addresses this gap through a teacher-conditioned readout criterion, whose added value is isolated by the matched-fit and auxiliary-objective controls below.
1. Existing evidence. Sections 3–4 distinguish the inherited trailing-query architecture from our contribution: selecting a shared readout within the frozen model’s realizable attention family. The score loss supervises the output, while TCD selects the internal context-attention readout. Table 3 tests whether the location of this criterion matters.
The contribution is the criterion and its placement within the frozen attention interface; at matched teacher fit, the context-attention formulation outperforms applying the same criterion to the full state.
2. Additional experiment. To distinguish this contribution from the query-based readout and prompt-tuning approaches it builds on, we compare the following methods with identical supervision, input representations, training budgets, and validation-only model selection.
(K+1)d + d + 1(K+1)d + d + 1(K+1)d + d + 1(K+1)d + d + 1Prompt tuning, prefix tuning, and the plain trailing-query readout perform comparably, so the inherited parameter-efficient design alone does not account for the result. The gain comes from applying TCD to the context-attention branch, at no additional parameter cost.
Reference
[1] DriveJudge: Rethinking Autonomous Driving Evaluation with Vision-Language Models. arXiv preprint, 2026.
[2] DriveCritic: Towards Context-Aware, Human-Aligned Evaluation for Autonomous Driving with Vision-Language Models. ICRA, 2026.
1.2 Advantage over simpler supervised representation regularizers
1. Existing evidence. Table 3 and Appendix F.6 compare readouts under the same architecture and matched training protocol. In the permutation control, the prediction loss retains the correct labels; only the labels used by TCD are shuffled.
The gain depends on correctly conditioned supervision rather than improved teacher fit.
2. Additional experiment. We replace only the auxiliary objective while keeping the architecture, training steps, seeds, and hyperparameter-search budget fixed. Signal-only alignment uses normalized latents to prevent gains from trivial scale inflation; the auxiliary regression head adds only a linear layer.
TCD outperforms auxiliary regression, signal alignment, and target-independent whitening at comparable training cost. Combining teacher-conditioned signal alignment with residual whitening yields the strongest external alignment.