acceptodds
Withdrawn from ICLR 2027

Chain-of-Attention: Field-Verifiable Reasoning for Traffic Vision–Language Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Traffic vision–language models can produce plausible answers without revealing when an event occurs, which actors are involved, or what visual evidence supports the explanation. We introduce Chain-of-Attention (COA), a phase-indexed record that binds five traffic phases to selected frames, pedestrian and vehicle boxes, and an event-level account. These explicit fields support separate training, scoring, reward optimization, and stress testing. On 48 fixed-camera WTS validation scenes, supervised training raises pedestrian IoU from .014 to .406 and vehicle IoU from .035 to .570 under the field-level evaluation with uniform sampling and no phase labels. Reward optimization preserves this grounding and improves blinded ratings of visual factuality, causal support, and evidence consistency. A phase-free control shows that supervision balance strongly affects reliable structured output. Balancing full-record and auxiliary examples raises strict schema completion from 3/48 to 15/48; COA reaches 16/48 under its corresponding strict schema. This experiment identifies task mixture as a major source of format failure, while COA provides an explicit interface for training and auditing evidence-bound predictions. We also expose a temporal shortcut: equal-per-phase sampling makes phase identity recoverable from input rank. Replacing images across 47 paired scenes collapses actor IoU while leaving the temporal score essentially unchanged, showing that temporal structure alone does not establish visual grounding.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
vision-language models, traffic video understanding, spatiotemporal grounding, structured reasoning, verifiable reward optimization, shortcut-aware evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_1dkaOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes Chain-of-Attention (COA), a structured way for VLMs to explain what is happening in a traffic scene. Instead of only giving an answer, the model identifies important frames, locates pedestrians and vehicles, and explains how the event develops over time. Authors show that their proposed framework greatly improves object localization and explanation quality.

Critical strengths

The reviewer likes the diagnostic focus of the paper, that instead of merely reporting accuracy scores, the authors actively test whether the model is really using the video. For example in Table 6, when they replace the actual scene images with images from a different scene, pedestrian/vehicle localization collapses, yet the temporal score stays almost unchanged (.887 → .888). That is a good experiment because it shows that a seemingly good temporal metric can be “cheated” using ordering cues rather than visual understanding. This also matches the reviewer's own experiences.

Critical weaknesses

The basic COA idea is not fundamentally new. [1] is particularly close: it explicitly grounds video reasoning using timestamps + objects + bounding boxes + reasoning, and trains with rewards for answer accuracy, temporal alignment, and spatial precision. That overlaps heavily with COA's “when + where + explanation” pitch. Prior works like [2-4] already use RL to jointly optimize temporal grounding, spatial grounding, and structured output format in videos. The differences are only on the traffic scene application. The reviewer doesn’t see the contribution of the paper to be of ICLR standards.

[1] Open-o3-Video: Grounded Video Reasoning with Explicit Spatio-Temporal Evidence. ICML 2026
[2] STVG-R1: Incentivizing Instance-Level Reasoning and Grounding in Videos via Reinforcement Learning, ICLR 2026
[3] Visual-RFT: Visual Reinforcement Fine-Tuning, ICCV 2025
[4] Grounded-VideoLLM: Sharpening Fine-grained Temporal Grounding in Video Large Language Models, EMNLP 2025

Questions for authors

Why is strict output validity so low, and how much of that is actually a modeling problem? Any analysis on this?

Reviewer_cjEdOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Chain-of-Attention (COA) for traffic video reasoning. It links five event phases to displayed frame IDs and pedestrian and vehicle boxes. The output also includes answers, an event account, and anchor information. Qwen3-VL-8B is trained with LoRA SFT and then GRPO with field rewards. Experiments use 95 training and 48 validation scenes from WTS. SFT improves localization over the base model. Human ratings favor GRPO, while its automatic grounding gains have confidence intervals crossing zero. Flat-format controls examine record completion. Sampling and pixel-replacement tests expose shortcuts in temporal evaluation.

Critical strengths
  1. The shortcut audit is useful. In 47 paired scenes, replacing pixels reduces GRPO pedestrian/vehicle IoU from .425/.621 to .013/.042. The temporal score changes only from .887 to .888 (Table 6). Questions, IDs, and annotations stay fixed. Observation 2 also explains exact phase recovery from rank under equal-per-phase sampling. These results show why temporal scores alone cannot establish visual grounding.

  2. The blinded human comparison supports a benefit from GRPO beyond automatic scores. Scene-level preferences are 35 for GRPO, one for SFT, and 12 ties. Visual factuality, causal support, and evidence consistency improve, with positive paired confidence intervals (Table 5). Two raters score anonymized outputs in randomized order. This is useful evidence for the explanation-quality claim.

Critical weaknesses
  1. Complete-record reliablity is low for COA-SFT. Only 16/48 outputs pass the strict audit (Table 3). COA does improve over Natural Flat at 3/48. Task-balanced Flat reaches 15/48 and has higher point estimates on the strict semantic metrics. These results do not establish equivalence or superiority. However, a task-balanced COA run is missing. The added value of the phase format under balanced training remains unclear. Table 1 scores recoverable fields without requiring a complete record, so its gains do not resolve this concern.

  2. The evaluation lacks independent confirmation. The same 48 validation scenes are used for GRPO checkpoint selection and reported evaluation. The SFT checkpoint selection rule is not recoverable (Sec. 4). The strict COA audit was specified after some output inspection, then frozen before full-corpus aggregation. Flat controls use one seed (Sec. 7). The paper discloses these limits. Still, the reported results cannot establish performance on scenes untouched by model selection and evaluation design.

Questions for authors
  1. Can you provide a task-balanced COA versus task-balanced Flat comparision with matched training and decoding budgets? Please report both field-level and complete-record results. This would help isolate the contribution of the phase format.

  2. Can you evaluate fixed checkpoints and parsers on scenes not used for checkpoint selection or audit design? This would directly address the second weakness.

  3. What are the strict complete-record validity and invalid-as-zero metrics for SFT+GRPO? Table 3 reports COA-SFT, so the effect of GRPO on record completion is unclear.

  4. Were the two primary raters authors or independent evaluators? Please also report output lengths for SFT and GRPO. These details would help interpret the human preference results.

Reviewer_6YHLOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

COA introduces a structured output format linking five traffic phases to selected frames, actor boxes and an event. The paper studies SFT and GRPO on WTS dataset, compares phase-indexed and Flat outputs, and studies whether temporal scores depend on visual evidence. SFT substantially improves spatial grounding, while GRPO’s clearest gains appear in blinded human evaluation. However, the benefit of COA’s phase structure is not established.

Critical strengths
  • The paper openly reports checkpoint selection, post-hoc evaluation changes and limitations. The image-replacement experiment is a useful diagnostic: actor localization collapses while the temporal score barely moves (Table 6), which shows that temporal structure and visual grounding need separate audits. I would like to see this kind of check adopted more widely, and it is why I do not recommend clear rejection.

  • The blinded human evaluation supports the claim that GRPO improves explanation quality, with 35 scene-level preferences for GRPO, one for SFT and 12 ties

Critical weaknesses
  • The benefit of COA’s phase structure is not convincingly established. The large Base-to-SFT gains do not isolate the contribution of phase indexing, while the GRPO result primarily concerns human-rated explanation quality. Under the strict audit, Balanced Flat achieves 15/48 valid records versus COA’s 16/48 and has higher semantic point estimates, including pedestrian IoU (.224 versus .146), although these differences are not established as significant.The balancing experiment creates a training-mixture confusion, which is useful, but on its own it is a limited contribution. A direct demonstration of improved evidence-consistency checking or error detection is needed to clarify what COA adds.

  • The final model’s complete-record reliability remains unknown. Table 3 applies the strict audit only to COA-SFT, which produces valid records on 16/48 scenes. Field-level accuracy and human preference do not establish that the final model reliably produces complete, correct evidence records. Critical-moment selection also remains weak: its field-level AFA@15 is .188, equal to the base model, against a sampler oracle of .979. These results limit the demonstrated usefulness of the proposed evidence-binding interface.

  • The principal findings lack independent confirmation and have limited scope. The same 48 WTS validation scenes are used for GRPO checkpoint selection and headline evaluation, and the SFT selection rule was not retained. The Flat comparison lacks training-seed replication. Although the paper includes a separate three-seed reward intervention and cross-backbone temporal probes, neither of these establishes the robustness or usefulness of COA. The authors disclose these limitations, but the evidence remains insufficient to support a general advantage beyond this setting.

  • Important evaluation and reproducibility details remain unclear. The historical parser defect is acknowledged without much details. The training frame-sampling protocol, verbatim prompts and complete COA/Flat output examples are insufficiently specified. No quantitative quality audit is reported for the GPT-4o-structured targets.

Questions for authors
  1. What full-record/auxiliary mixture did COA-SFT actually receive, and which frame samplers generated the SFT and GRPO training inputs?
  2. What are the strict whole-record validity and invalid-as-zero semantic scores for the final SFT+GRPO model? Table 3 reports these only for COA-SFT.
  3. What exactly was the historical parser defect, and did it affect recovered fields, parse flags or reported semantic metrics?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.