- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper probes whether vision-language driving models revise their action decision when a small change in another road user's behavior makes the previously appropriate ego action unsafe. From real nuScenes scenes, it builds 150 reviewed video pairs using four atomic interventions on existing objects. Humans independently assign KEEP, SLOW, or BRAKE_OR_STOP to each original and counterfactual video. Five models are evaluated (DriveTA, Qwen2.5-VL-7B, LLaVA-Video-7B, Qwen3-VL-8B, LLaVA-OneVision-7B) with paired metrics.
Critical strengths
- The core design insight is right and clearly articulated: whether an intervention should change the ego decision depends on the complete traffic context, so the intervention type cannot serve as the answer label. Deriving the required response from two independently reviewed action labels, including retention cases, is exactly the right formulation, and the Always KEEP diagnostic cleanly exposes the label-prior problem (20.67% PairAcc with 0% DIR-strict).
- The evaluation is well controlled: three semantically equivalent option orders are averaged, the two videos are scored in separate forward passes, and the evidence-removal controls (no visual input drives Qwen2.5-VL's DIR-strict to zero; final-frame and temporal-position ablations) separate access to evidence from the ability to use it. The adaptation study with a source-scene-disjoint split and a clean class-recall breakdown (BRAKE/STOP 0% to 50.88%, SLOW 66.67% to 0%) is a model of how to report a negative result.
Critical weaknesses
- The benchmark is small and the reliability of the gold labels is undocumented. The central number (DIR-strict at most 13.16%) rests on 76 change-required pairs, the less-conservative analysis on 11 pairs, and per-intervention claims on 14-55 pairs; no confidence intervals are reported. More importantly, the paper does not state how many annotators labeled each video, what inter-annotator agreement was on the ordinal three-way action space, or how disagreements were resolved. Since every gold relation derives from human judgments over 2.67-second clips, label noise could account for a meaningful share of the low model scores, and the "models fail to revise" interpretation is sensitive to it.
- "Fails to respond" is not fully disentangled from "fails to perceive." The no-visual control shows that models need visual input to respond, but there is no probe of whether models actually attend to the edited actor. Because the counterfactual videos come from a 1.3B-parameter generator (Wan2.1-T2V-1.3B via CompoSIA) with only qualitative review, subtle edit-realization failures or video artifacts could depress adjustment responses independently of decision-making ability. In addition, all evaluated models are open 7-8B VLMs plus one small adapted baseline; whether the finding holds for stronger or closed-loop driving models is untested, which makes the abstract-level claim ("VLMs" fail to revise) broader than the evidence.
Questions for authors
- How many annotators labeled each video, what was the inter-annotator agreement (e.g., kappa on the ordinal action space), and how were disagreements resolved? Are the 150 gold labels stable under re-annotation?
- Do the models actually perceive the edited interaction? Please add a perception probe (e.g., whether the model's free-form response or attention mentions the edited actor) and analyze whether adjustment failures correlate with edit-realization quality or the generation model used.
- With only 76 change-required pairs (11 of them less-conservative), what are the confidence intervals on DIR-strict and relation recall? Would the retention-bias conclusion hold if the benchmark were balanced toward adjustment-required pairs, e.g., by generating more forward-away and cut-in cases?
- Have you tested stronger models (e.g., larger closed VLMs, closed-loop planners) or a prompt that directs the model to identify the most decision-relevant change? The current conclusion is scoped to 7-8B open models.
Thank you for the clear answers and the additional annotation study. Reporting the exclusion counts, the scene-clustered intervals, and the seed variation for adaptation is helpful. My main concern about label reliability remains, though. The original labels came from a single annotator who was also an author and could see the paired video, while the paper says that human review "independently" assigns each action. The three new annotators agree with each other only moderately (α = 0.51), and the paired-relation agreement with the original labels is 56.7%–70.0%. I appreciate that the main trends hold under majority-vote labels, but I think the labeling protocol and its limits should be stated clearly in the main paper, and the benchmark would benefit from multi-annotator labels from the start. I will keep my score.