acceptodds
Under review as a conference paper at ICLR 2027

Bridging Perception, Behavior, and Interaction Reasoning in Roadside Traffic Scenes

Abstract

Urban traffic scenes involve dynamic multi-agent behaviors and interactions, yet existing roadside benchmarks emphasize agent-level perception or holistic event reasoning, leaving behavior evolution and multi-agent interactions insufficiently modeled and evaluated. We introduce TRACE-Bench, a benchmark for roadside Traffic Reasoning through Agent-Centric Evolution. TRACE-Bench adopts a hierarchical annotation framework linking scene context, agent perception, behavior evolution, directed interactions, and causal reasoning. Continuous agent motion is decomposed into temporally grounded behavior units, while interactions are represented as directed influence–response relations. Built from real-world roadside RGB-Event recordings, TRACE-Bench comprises 201 finely annotated video sequences and 22,807 evidence-grounded QA pairs. We further propose an Event-Guided Sparse Multimodal framework that progressively reduces redundancy and fuses complementary RGB-Event information for efficient fine-grained traffic reasoning. Extensive experiments reveal limitations of existing MLLMs in behavior understanding and interaction reasoning, while demonstrating that our framework improves traffic perception and understanding with reduced inference latency.

Primary area:
datasets and benchmarks
Keywords:
Roadside Traffic Understanding, Multimodal Large Language Models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_jCTXOpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces TRACE-Bench, a dataset for testing whether VLMs can understand roadside traffic videos beyond simply detecting cars and pedestrians. It evaluates whether models can follow how road users move, interact, and reason about possible outcomes. The authors also propose a method that uses event-camera signals to select important video frames and regions, reducing unnecessary computation while improving traffic understanding. Experiments show that vision-language models still struggle with complex interactions.

Critical strengths
  • The reviewer really likes the high quality presentation of the paper. Great work authors in putting together the manuscript and writing (of course with some need for improvements).
  • The strongest part for the reviewer is TRACE-Bench’s explicit reasoning chain from perception → behavior → directed interaction, with each QA tied back to structured evidence rather than only a final answer. This diagnostic capability is much more useful for a benchmark than a simple overall accuracy metric. Evaluating how well responses are grounded reveals far more about whether and how these VLMs hallucinate.
Critical weaknesses
  • So one thing most critical to the reviewer is that the authors label directed interactions as one agent inducing another agent’s response, BUT the dataset is observational roadside video. That does not establish causality: both agents could be responding for example to a traffic light, road geometry, another vehicle, or some other latent factor. The counterfactual task has the same problem: there is no true observed ground truth for what would have happened under the hypothetical change. In the reviewer’s opinion, this would hamper the “causal reasoning” claim of the paper.
  • The effective dataset is also small: 201 videos. The headline “22,807 QA pairs” sounds large, but they come from only 201 sequences and roughly 1.4K agents. Moreover, questions are populated from manually designed templates. That means many examples are correlated and the effective visual diversity is much smaller than 22.8K suggests. This becomes particularly problematic when small performance gains are reported without video-level confidence intervals.
  • Authors argue that event data helps identify informative frames and regions, but it does not compare against strong RGB-only motion cues such as optical flow, frame differences, background subtraction, etc. What’s the benefit from event sensing specifically, rather than simply from having any motion-based importance signal?
Questions for authors
  • In Table 8-9, the proposed model gets 83.05% accuracy at night vs 71.96% during daytime. The same pattern appears for the RGB and Event-only baselines too. That is interesting because the paper motivates event cameras partly as being especially useful under difficult illumination. But these numbers suggest the night subset may simply be easier overall, so the day/night comparison is confounded by dataset difficulty. What do the authors think of this analysis?
  • Qwen family models (2.5+) have used dynamic sampling for video processing. Why not use that, and come up with your own sampling method? Any comparisons?
5 comments

We sincerely appreciate your highly positive assessment of the manuscript's presentation and the value of TRACE-Bench, as well as your insightful and constructive comments. We have carefully considered your concerns and provide our point-by-point responses below.


W1: Causal Validity of Directed Interactions and Counterfactual Reasoning.

For directed interaction annotation, we do not infer interactions solely from the temporal order of two agents' behaviors. Instead, we require sufficient visual evidence of an influence–response relationship, considering their relative spatial positions, behavioral changes, temporal order, and surrounding traffic context. An interaction is annotated only when these cues collectively support a plausible interpretation. Cases with insufficient visual evidence or where behavioral changes cannot be distinguished from responses to shared environmental factors are excluded. We use roadside videos because their wider field of view provides more comprehensive observations of traffic participants and their surroundings, supporting more informed interaction judgments.

Although annotating interactions in complex traffic scenes is challenging, understanding influence–response relationships is essential for analyzing traffic dynamics. We therefore believe that constructing such annotations under explicit evidence constraints is both meaningful and necessary. To ensure reliability, we establish consistent interaction annotation criteria and independently review interaction labels, focusing on whether the interacting agents, response behaviors, and spatiotemporal relationships are supported by observable evidence.

For counterfactual questions, the task evaluates whether a model can identify the potential consequence or safety-related mechanism most consistent with the hypothetical condition and available visual evidence. Answer uniqueness is defined within the given hypothesis and answer options, rather than implying a uniquely determined or verifiable real-world counterfactual outcome. All counterfactual questions undergo independent review, and cases with ambiguous hypotheses or multiple plausible answers are revised or excluded.

We have revised the manuscript to avoid equating evidence-grounded interaction understanding with causal inference, and clarified that the counterfactual task assesses plausibility under specified hypothetical conditions rather than estimating actual causal effects. The specific revisions are in "Abstract" on page 1, "Introduction" on pages 2 and 3, "A.3.3 QA Construction" and "A.3.4 Quality Control" on pages 20 and 21 of the revised manuscript.


W2: Visual Diversity, Correlated QA Samples, and Statistical Reliability.

The primary contribution of TRACE-Bench is not large-scale scenario coverage, but a fine-grained evaluation framework with explicit spatiotemporal evidence and cross-level associations, addressing the limitations of existing benchmarks in evaluating continuous behavior evolution and directed interaction understanding. This requires constructing interconnected QA pairs from agent, behavior, and interaction annotations within the same video, enabling systematic evaluation, as well as cross-level error attribution. Thus, the 22,807 QA pairs primarily reflect the granularity and scope of the evaluation tasks, offering research value distinct from simply increasing the number of videos or environmental categories.

The fine-grained annotations in TRACE-Bench require substantial human effort and rigorous verification, ensuring high-quality structured annotations while limiting dataset scale. We will clarify this limitation in the revised manuscript and explore more efficient annotation methods to expand dataset scale and scenario coverage in the future.

To assess the statistical reliability of the performance gains, we conducted a paired, clip-level bootstrap analysis on the validation set, using 10,000 resamples with replacement at the video-clip level. The resulting 95% confidence interval for the accuracy improvement over the RGB-only Qwen3-VL-4B baseline is [0.26, 1.44], providing statistical support for the observed gain. Furthermore, compared with the RGB-only baseline, our method reduces inference latency by 22.2% and FLOPs by 60.5%, demonstrating a better accuracy–efficiency trade-off.

The specific revisions are in "Limitations and Future Work" on page 10 and "Fine-tuned Performance" on page 9 of the revised manuscript.

W3: Event-Specific Benefits over RGB-Based Motion Cues.

To assess the benefits of event signals over conventional RGB-based motion cues, we conduct additional ablations by replacing the event signals in our sparsification method with RGB frame differences and optical flow estimated by RAFT, while keeping all other settings unchanged.

As shown in the table below, the event-based method achieves 78.77 accuracy, outperforming frame differences and optical flow by 2.12 and 2.19, respectively, while yielding the lowest spatiotemporal localization error. Its inference latency is comparable to that of frame differences and substantially lower than that of RAFT, which introduces additional computational overhead for optical flow estimation. These results demonstrate that the high temporal resolution of event sensing enables finer-grained motion representation, facilitating more effective selection of motion-salient frames and regions.

MethodAcc. ↑T-MAE ↓ST Err. ↓BERT-F1 ↑Latency (s) ↓
Frame Difference76.654.270.51980.27411.79
Optical Flow (RAFT)76.584.250.51250.26972.45
Event (Ours)78.774.250.50090.27521.82

The specific revisions are on "Benefits of Event Signals for Spatiotemporal Sparsification" on page 18 of the revised manuscript.


Q1: Day–Night Performance Discrepancy and Event Camera Advantages under Low-Light Conditions.

We thank the reviewer for this insightful analysis. First, higher nighttime accuracy does not necessarily indicate that the nighttime subset is easier across all tasks. As shown in Tables 8–9, although our method achieves higher accuracy at night, its spatial and spatiotemporal localization errors increase (S-L1: 0.1390 vs. 0.1133; ST Err.: 0.5402 vs. 0.4217), while BERT-F1 for open-ended QA decreases (0.2449 vs. 0.3047). Similar trends are observed for the RGB-only and Event-only baselines. These results suggest that day–night performance differences vary across tasks and may reflect both illumination conditions and differences in scene difficulty between the two subsets.

Nevertheless, comparisons within the same evaluation subsets provide evidence of the potential complementary value of event data under low-light conditions. Specifically, on the daytime subset, RGB-only outperforms Event-only in BERT-F1 and BLEU-4. However, on the nighttime subset, Event-only surpasses RGB-only on both metrics (0.2457 vs. 0.2404 and 0.0466 vs. 0.0431). This shift in relative performance suggests that event data can provide potential complementary information under low-light conditions. We will clarify the interpretation and scope of the day–night evaluation in the revised manuscript.

The specific revisions are on "A.2.2 Performance Under Different Illumination Conditions" on page 16 of the revised manuscript.


Q2: Event-Guided Sampling vs. Qwen's Dynamic Video Sampling.

We would like to clarify that Qwen's native dynamic sampling mechanism selects frames at approximately uniform temporal intervals according to the different FPS or frame budget. Therefore, it does not explicitly prioritize frames based on their motion content. In contrast, our Event-Guided Frame Sampling (EGFS) leverages event activity and temporal novelty to identify dynamically informative moments, while retaining uniformly sampled frames to preserve global temporal context. Furthermore, in Section 5.3 (Temporal Sampling Strategy), we directly compare EGFS with the uniform temporal sampling strategy used in Qwen under the same frame budget. The results show that EGFS achieves higher accuracy, demonstrating the effectiveness of event-guided selection in preserving informative temporal cues for traffic understanding.


We have uploaded the revised manuscript, incorporating the additional results and clarifications discussed above. We welcome any further questions or suggestions and would be happy to provide additional clarification. We look forward to your feedback.

Reviewer_jCTXOpenReview ↗

Response to rebuttal

Hi Authors,

First of all, thank you for your hard work.

  1. Given your response to W3, the reviewer would always choose "Frame Difference" as it is easy to implement and scale. The reviewer doesn't see the value in Event (Ours) for the spatiotemporal sparsification based the triviality of the gains.

  2. For the Qwen-style sampling, the reviewer thinks it is beyond "simple" uniform sampling. "Dynamic" and "uniform" won't go hand in hand, right? Even if conceding that "uniform" is approximating the "dynamic" sampling, the gains are again trivial (Table 5).

  3. The above two points brings the reviewer to the main point which other reviewers are raising as well: The reviewer believes the work is being limited by the number of video samples in the dataset. In the response to Reviewer udWP's comment, the numbers do not present the scale of benchmark needed to make strong conclusions. This resonates from the reviewer's own experience as well.

Given the above points, the reviewer maintains the rating of "Weak Reject". The reviewer will not choose "Clear Reject" as the paper is still in the right direction. Hope the comments help!

Reviewer_udWPOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces TRACE-Bench (Traffic Reasoning through Agent-Centric Evolution), a new benchmark featuring a structured 5-level cognitive hierarchy: static scene, agent perception, behavior, directed interaction, and counterfactual/hallucination reasoning. The dataset comprises 201 video clips (10–30 seconds each), yielding 22,807 QA pairs. Additionally, the authors propose an Event-Guided Sparse Multimodal framework built on Qwen3-VL-4B that fuses RGB frames with event data to eliminate token redundancy and reduce inference latency. The benchmark evaluates both general MLLMs and domain-specific fine-tuned models.

Critical strengths
  1. Bridges gaps in existing benchmarks by integrating scene context, agent perception, behavioral evolution, directed interactions, and causal reasoning.
  2. Introduces a novel multimodal fusion strategy combining RGB frames and event data via the proposed EFGS, DRTS, and AERF modules to reduce inference latency by filtering redundant spatio-temporal tokens.
  3. Evaluates domain-specific models (TUMTraffic-Qwen-7B and EventGPT-7B) for event-based multimodal understanding alongside general MLLMs.
Critical weaknesses
  1. Limited Dataset Scale and Diversity: The dataset contains only 201 clips (10–35 seconds each, totalling ~33–100 minutes), which seems insufficient to capture realistic traffic variability. Furthermore, Figure 3 highlights a narrow environmental distribution, with high structural similarity across scenes (e.g., "Urban Expressway" vs. "Overpass", and "Merging Area" vs. "Diverging Area"), raising concerns about scenario diversity.
  2. Lack of Evaluation in Challenging Conditions: The benchmark omits common adverse driving conditions (e.g., low light, fog, rain, glare), where vision models typically fail. Moreover, the single day/night comparison in Figures 6–7 and Tables 8–9 reveals an anomaly: the model performs significantly better at night than during the day, which contradicts typical vision challenges and suggests potential dataset bias.
  3. Missing Baselines and Unexplained Anomalies: Despite claiming to benchmark representative MLLMs, several key video/multimodal baselines are missing (e.g., LLaVA-OneVision, LLaVA-Video, InternVideo2, VILA, Phi-3.5-Vision). Furthermore, the authors do not adequately explain why larger models underperform smaller ones (e.g., 8B vs. 4B in Intern and Qwen series), nor why fine-tuned domain-specific models underperform general MLLMs.
  4. Flawed Evaluation Metric for Binary Tasks: Evaluating binary premise verification using raw accuracy is compromised by affirmative bias (e.g., VQAv2 [1]; POPE [2]; HallusionBench [3]), where MLLMs tend to answer "Yes" (>85%). F1-score, Precision-Recall, or paired-consistency metrics should be used instead of raw accuracy to properly assess hallucinations and counterfactual reasoning.

Minor:

  1. Inflated Baseline Reference: The reported latency reduction from 3.95s to 1.82s in Section 5.3 is misleading, as the 3.95s reference relies on an unoptimised dual-stream baseline (RGB-Event Add*). Compared to the standard single-camera Qwen3-VL-4B baseline (2.34s), the actual reduction is only 22.2%. Inconsistent usage between RGB-Event Add* (in the ablation study) and RGB-Event Add (elsewhere) causes confusion and appears to artificially inflate the baseline latency.

[1] Y. Goyal, T. Khot, D. Summers-Stay, D. Batra and D. Parikh, "Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering," 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[2] Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. 2023. Evaluating Object Hallucination in Large Vision-Language Models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing.

[3] T. Guan et al., "Hallusionbench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Models," 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

Questions for authors
  1. Given the well-documented affirmative bias of MLLMs (>85% "Yes"), could the authors provide F1-scores, Precision/Recall, or paired-consistency metrics (as standardised in POPE/HallusionBench) for binary premise verification instead of relying solely on raw accuracy?
  2. Could the authors include evaluations on missing representative video MLLMs (e.g., LLaVA-OneVision, LLaVA-Video, InternVideo2, VILA), and explain both the counterintuitive performance drops in larger model scales (4B vs. 8B) and why domain-specific fine-tuned models underperform general MLLMs?
  3. What explains the counterintuitive result where models perform significantly better at night than during the day (Tables 8–9), and how do the authors justify the benchmark's real-world coverage given the limited scale (201 clips) and lack of adverse conditions (fog, rain, glare)? Could the authors provide a comprehensive breakdown and quantitative distribution of the driving scenarios, specifically regarding environmental conditions (e.g., sample counts across weather types like rain/fog, lighting levels, glare, road geometries, and traffic density)?
  4. Given the limited number of QA/prompt templates that overlap with the test set without rephrasing (paraphrasing while preserving semantic meaning), could the authors address the concern that fine-tuned models are merely memorizing these fixed templates rather than performing genuine traffic reasoning?
  5. Could the authors clarify the exact implementation differences between RGB-Event Add* (3.95s) and RGB-Event Add, and report the actual wall-clock speedup and FLOPs reduction compared directly to the standard, optimized single-camera Qwen3-VL-4B baseline (2.34s)?
7 comments

We sincerely appreciate your positive assessment of our proposed benchmark and method, as well as your constructive comments. We have carefully considered your concerns and provide our point-by-point responses below.


W1 & W2 & Q3: Dataset Scale, Scenario Diversity, and Day/Night Performance Differences.

Dataset Scale and Scenario Diversity. The primary goal of TRACE-Bench is not to achieve exhaustive coverage of all road and environmental conditions, but to address the limitations of existing benchmarks in evaluating continuous behavior evolution and directed interaction understanding. To this end, TRACE-Bench establishes a fine-grained evaluation framework with explicit spatiotemporal evidence and cross-level associations. This enables systematic evaluation from agent perception to behavior understanding and interaction reasoning (Section 5.2), as well as cross-level error attribution (Appendix A.2.1). Such fine-grained evaluation is the main contribution of TRACE-Bench, offering research value distinct from simply increasing the number of videos or environmental categories.

We further analyze the dataset distribution across road types, topologies, lighting conditions, and off-peak/peak traffic periods. The dataset was collected from 14 sites with different road topologies, covering six representative urban traffic scenarios: urban arterial roads (56 clips, 3 sites), overpasses (36 clips, 2 sites), merging areas (35 clips, 1 site), intersections (28 clips, 4 sites), urban expressways (26 clips, 3 sites), and diverging areas (20 clips, 1 site). Importantly, similar road geometries do not necessarily imply similar traffic characteristics. For example, merging and diverging areas involve vehicles entering and exiting the main roadway, respectively, leading to different movement paths, lane-changing demands, and interactions with surrounding vehicles, which may produce distinct behavior sequences and interaction patterns.

Regarding lighting conditions, the dataset contains 105 daytime and 96 nighttime clips, corresponding to 11,741 and 11,066 QA pairs, respectively. Based on the peak-hour definitions provided by local traffic authorities, the dataset also includes 90 off-peak and 111 peak-hour clips, corresponding to 10,411 and 12,396 QA pairs, respectively. These statistics demonstrate that TRACE-Bench covers real-world traffic scenarios across different road topologies and traffic periods, with relatively balanced distributions across daytime/nighttime and off-peak/peak-hour conditions.

Day–Night Performance Differences. Higher nighttime accuracy does not necessarily indicate that the nighttime subset is easier across all tasks. As shown in Tables 8–9, although our method achieves higher accuracy at night, its spatial and spatiotemporal localization errors increase (S-L1: 0.1390 vs. 0.1133; ST Err.: 0.5402 vs. 0.4217), while BERT-F1 for open-ended QA decreases (0.2449 vs. 0.3047). Similar trends are observed for the RGB-only and Event-only baselines. These results indicate that day–night performance differences vary across tasks and may reflect both illumination conditions and differences in scene difficulty between the two subsets.

Limitations and Future Work. The fine-grained annotations in TRACE-Bench require substantial human effort and rigorous verification. While this supports high-quality structured annotations, it also limits dataset scale. In future work, we plan to explore more efficient annotation methods to expand dataset scale and scenario coverage while maintaining annotation quality.

The specific revisions are in "A.3.1 Dataset Statistics" on page 19 and "Limitations and Future Work" on page 10 of the revised manuscript.

W3 & Q2: Missing MLLM Baselines and Model Performance Discrepancies.

ModelAcc. ↑T-MAE ↓S-L1 ↓ST Err. ↓BERT-F1 ↑ROUGE-L ↑BLEU-4 ↑CIDEr ↑
InternVideo2-8B26.708.360.40671.73210.09290.19350.04130.0603
VILA-1.5-8B37.969.460.88111.73210.22580.23730.02980.0268
LLaVA-Video-7B57.954.420.34720.53650.24760.27610.03960.1785
LLaVA-OneVision-7B51.104.510.34450.52340.21420.23620.01430.0849

We thank the reviewer for the valuable suggestions. We have additionally evaluated four representative video MLLMs, including LLaVA-OneVision-7B, LLaVA-Video-7B, InternVideo2-8B, and VILA-1.5-8B, with results presented in the table above. LLaVA-Video and LLaVA-OneVision achieve accuracies of 57.95% and 51.10%, respectively, while InternVideo2 and VILA achieve 26.70% and 37.96%. One possible reason for the relatively poor performance of InternVideo2 and VILA is their limited fine-grained spatiotemporal grounding capabilities. In particular, these models frequently produce invalid spatiotemporal coordinates during evaluation, which may further affect their agent-level behavior understanding and interaction reasoning performance.

Regarding model scaling, we clarify that the performance decline with increasing model size is mainly observed in the zero-shot evaluation of Qwen3-VL and InternVL3.5. However, the 4B variants do not consistently outperform their 8B counterparts across all tasks. For example, Qwen3-VL-8B achieves better spatial localization (S-L1: 0.2657 vs. 0.4137) and temporal localization (T-MAE: 4.40 vs. 4.64) than Qwen3-VL-4B. These results indicate that performance trends vary across evaluation dimensions, and increasing model size does not necessarily yield consistent improvements across traffic understanding tasks. Performance may also depend on pretrained representations and task-specific adaptation capabilities.

Regarding specialized models versus general-purpose MLLMs, we clarify that both TUMTraffic-Qwen-7B and Event-GPT-7B were fine-tuned on TRACE-Bench to ensure consistent evaluation settings. TUMTraffic-Qwen is designed for traffic video understanding, whereas Event-GPT specializes in event-based vision rather than the traffic domain. Despite their specialized designs, both models underperform the fine-tuned Qwen3-VL baseline. This difference may be attributed to variations in backbone architectures, visual representations, and multimodal alignment. Since TRACE-Bench jointly evaluates agent perception, behavior evolution, and directed interaction reasoning, general-purpose MLLMs such as Qwen3-VL may offer stronger adaptability across these diverse tasks.

The specific revisions are in "Evaluated Models" on page 7 and "5.2 Main Results" on pages 8 and 9 of the revised manuscript.

W4&Q1: Affirmative Bias and Binary Evaluation Metrics.

To further assess potential affirmative bias, we additionally report Precision, Recall, F1-score, and the proportion of “Yes” predictions for binary premise verification. Our method achieves 89.05% Accuracy, 96.31% Precision, 80.68% Recall, and 87.80% F1-score. Moreover, the model predicts “Yes” in only 40.94% of cases, compared with 48.87% in the ground-truth answers, indicating no excessive affirmative bias. In particular, the high Precision demonstrates that positive predictions are highly reliable, while the F1-score further indicates that the performance is not merely driven by a bias toward one answer category.

The specific revisions are on "A.2.5 Affirmative Bias Analysis in Binary Premise Verification" on page 18 of the revised manuscript.


W5&Q5: Baseline Definitions and Efficiency Comparison.

We thank the reviewer for this comment. We would like to clarify the distinction between RGB-Event Add* and RGB-Event Add. RGB-Event Add* directly combines RGB and event features through addition without spatiotemporal sparsification and serves as the reference for evaluating individual components in the ablation study (Section 5.3 and Table 4). In contrast, RGB-Event Add incorporates spatiotemporal sparsification and is used as a baseline for evaluating the proposed fusion strategy under comparable computational settings (Section 5.2 and Table 2).

The efficiency improvements are measured against the standard RGB-only Qwen3-VL-4B baseline under identical hardware and inference settings, as already reported in Section 5.2 and Table 3. Specifically, our method reduces wall-clock inference latency from 2.34 s to 1.82 s (22.2% reduction, 1.29× speedup) and TFLOPs from 195.69 to 77.30 (60.5% reduction).

We have clarified the definitions and purposes of these baselines in the revised manuscript to avoid ambiguity. The specific revisions are in "Component Ablation" on page 9 of the revised manuscript.


Q4: QA Template Memorization.

The predefined QA templates are designed to ensure accurate alignment between questions and structured annotations. Although the training and validation sets may share template structures, the correct answers depend on the specific traffic agents, behavior evolution, and interactions observed in different videos. Furthermore, our existing Text-only ablation (Appendix A.2.4 and Table 10) shows that the model achieves only 33.17 accuracy without visual input, substantially lower than the 78.77 achieved by the full RGB–Event model. This indicates that question text and language priors alone are insufficient for effective traffic understanding, highlighting the importance of visual evidence.

To further assess whether the model relies on fixed question formulations, we conducted a semantics-preserving paraphrasing robustness experiment. Specifically, we used GPT-6 to paraphrase validation questions and answer options with diverse wording while strictly preserving their original meanings, agent identities, spatiotemporal information, and correct answers. All paraphrases were reviewed by human experts. Without additional fine-tuning, our method achieves 77.93 accuracy on the paraphrased evaluation set, only 0.84 below the original validation accuracy of 78.77. These results demonstrate the model's robustness to semantically equivalent formulations and suggest that its performance is not primarily driven by memorization of fixed QA templates.


We have uploaded the revised manuscript, incorporating the additional results and clarifications discussed above. We welcome any further questions or suggestions and would be happy to provide additional clarification. We look forward to your feedback.

Reviewer_MFcNOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces TRACE-Bench for roadside traffic understanding. It contains 201 RGB-Event clips and 22,807 QA pairs. The annotations link agent identities, timed behaviors, and directed interactions. The tasks cover perception, behavior, interaction, counterfactual reasoning, and hallucination. The paper also proposes event-guided frame selection, gated RGB-Event fusion, and token selection on Qwen3-VL-4B. Experiments cover eight zero-shot models and several fine-tuned baselines. The proposed model reaches 78.77% accuracy, compared with 78.13% for fine-tuned RGB-only Qwen3-VL-4B. Reported latency falls from 2.34 to 1.82 seconds.

Critical strengths
  1. The linked annotations support useful error analysis. Behavior units have time intervals, and interactions identify the initiating and affected agents. Table 7 tests interaction recognition after correct path-action recognition. RGB-only Qwen3-VL-4B still has 33.33% error in this setting. This gives the benchmark value beyond reporting one overall QA score.

  2. The RGB-Event setting is useful for studying traffic dynamics. The paper includes daytime and nighttime results. Section 5.2 reports a 4.65-point gain in directed interaction accuracy over the RGB-only model. Table 9 also shows that Event-only exceeds RGB-only on all four open-ended metrics at night. These results support further study of event data for this task.

Critical weaknesses
  1. Label reliability is not quantified. Appendix A.3.3 describes annotation guidelines, expert review, and averaging temporal boundaries from three annotators. However, it reports no agreement scores or boundary variation. Appendix A.3.2 explains how counterfactual questions change a local condition. Figure 1 gives an example, but the rule for selecting one correct hypothetical outcome is not clear. These gaps leave the reliablity of the benchmark's main labels uncertain. This is a central concern for a benchmark paper.

  2. The method comparison needs stronger evidence. The overall accuracy gain over RGB-only is 0.64 points (Table 2), with no repeated-run statistics or confidence intervals. Only a train/validation split is described (Sec. 3.3). Section 5.3 selects the 0.85/0.15 sampling mix from the reported results, but no separate final evaluation is described. Tables 5 and 6 test selection cues within the RGB-Event framework. They do not provide an RGB-only baseline with the same frame and token budget. The latency reduction is useful, but this missing comparision limits the claim that event guidance gives a better accuracy-cost trade-off.

Questions for authors
  1. What are the agreement scores for behavior and interaction labels, and the variation in temporal boundaries? What rule and checks establish a unique correct answer for counterfactual questions?

  2. Which split was used to select the settings in Tables 5, 10, and 11? Is there a separate final evaluation set that was not used for these choices?

  3. Can you report repeated-run results or confidence intervals resampled by clip for Ours and RGB-only? Please include overall accuracy and directed interaction accuracy.

  4. What accuracy and latency does RGB-only achieve with the same retained frame and token counts, using uniform frames and RGB-text token relevance?

  5. Will the RGB videos, event streams, and annotations be released? Under what license? What specific privacy processing was applied?

4 comments

We sincerely appreciate your highly positive assessment of our dataset and RGB–Event setting, as well as your constructive comments. We have carefully considered your concerns and provide our point-by-point responses below.


W1 & Q1: Annotation Reliability and Counterfactual Answer Validity

Annotation Agreement. We re-examined the raw annotations independently produced by three annotators and evaluated agreement on behavior segments, behavior categories, temporal boundaries, and directed interactions before expert adjudication and boundary aggregation. Since behavior annotation involves both temporal segmentation and semantic classification, we first matched behavior segments across annotators based on agent identity, behavior dimension, and temporal overlap. Three segments were considered a match only if all pairwise temporal IoUs were at least 0.5, with each segment assigned to at most one matched group.

Under these criteria, 89.3% of the original behavior segments were successfully matched across all three annotators, with the remaining differences mainly involving segment splitting, merging, or unmatched segments. Among matched segments, behavior category agreement reached 96.7%. The median pairwise absolute deviations in temporal boundaries were 0.32 s for start times and 0.38 s for end times, with corresponding 90th percentiles of 0.85 s and 0.96 s, indicating consistent temporal boundaries among matched segments. For directed interactions, the three annotators achieved 88.6% agreement on interaction existence. Among cases where all three identified an interaction, agreement reached 97.1% for initiator–affected agent direction and 98.0% for response behavior categories. These results indicate that disagreements primarily arise from behavior segmentation and interaction existence judgments, while identified interactions exhibit high consistency in direction and response behavior. Ambiguous cases are further reviewed and adjudicated by expert annotators to improve annotation reliability.

Counterfactual Answer Validity. The task evaluates whether a model can identify the potential consequence most consistent with the hypothetical condition and available visual evidence. During question construction, we modify only one behavior or response condition of a designated agent, while keeping the other observable contextual conditions unchanged. The correct answer must be supported by the visual evidence and specified hypothetical condition, while alternative options must be distinguishable based on road geometry, movement directions, interacting agents, or local safety mechanisms. To ensure answer uniqueness, two independent reviewers specifically examined the counterfactual QA pairs, checking the clarity of hypothetical conditions, the evidence supporting the correct answer, and the presence of multiple plausible options. Questions without a uniquely supported answer were revised or excluded. Thus, answer uniqueness is defined within the specified hypothesis and candidate options, rather than implying a uniquely determined real-world counterfactual outcome.

The specific revisions are on "A.3.3 QA Construction" and "A.3.4 Quality Control" on pages 20 and 21 of the revised manuscript.


W2 & Q2: Hyperparameter Selection and Evaluation Protocol.

TRACE-Bench uses a scene-disjoint training/validation split, without an additional independent test set. Nevertheless, the performance improvements are not restricted to the selected hyperparameter configuration. As shown in Table 5, all four event-guided/uniform sampling ratios (0.5/0.5–0.9/0.1) outperform the RGB-only baseline across multiple metrics. Similarly, Table 11 shows that varying the temporal pruning ratio from 25% to 75% (with 50% spatial pruning) consistently achieves higher accuracy (78.60%–78.77% vs. 78.13%) and lower inference latency (1.61–1.95 s vs. 2.34 s) than RGB-only. Other spatial pruning configurations also maintain competitive performance across multiple metrics with reduced latency. These results suggest that our method maintains a better accuracy–efficiency trade-off across a range of configurations rather than relying on a single optimum. Regarding Table 10, it examines how the temporal granularity of visual observations affects performance and latency. In the main comparison, all methods use 2 FPS for consistent frame sampling.

W2 & Q3: Statistical Significance of Performance Improvements.

Following your advice, to assess the statistical reliability of the observed improvements over RGB-only Qwen3-VL-4B, we conduct a paired clip-level bootstrap analysis on the validation set using 10,000 resamples with replacement at the video-clip level.

The resulting 95% confidence intervals for the improvements in overall accuracy and directed interaction accuracy are [0.26, 1.44] and [0.62, 9.32], respectively. Both intervals exclude zero, providing statistical support for the observed improvements under clip-level resampling. Furthermore, our method reduces inference latency by 22.2% and FLOPs by 60.5% compared with RGB-only Qwen3-VL-4B, demonstrating a better accuracy–efficiency trade-off.

The specific revisions are on "Fine-tuned Performance" on page 9 of the revised manuscript.


W2 & Q4: Comparison with RGB-only Baseline under Matched Computational Budgets.

Thank you for this valuable suggestion. We conduct an additional experiment comparing uniform frame sampling with RGB-text relevance-based token pruning against our EGFS + DRTS, using Qwen3-VL-RGB-4B as the baseline with identical retained frame and token counts. As shown in the table below, compared with Uniform + RGB-Text, our EGFS + DRTS improves accuracy by 0.55 and reduces spatiotemporal localization error, with comparable latency. These results demonstrate that our improvements arise not merely from reducing frames and tokens, but also from more effective frame and token selection.

MethodAcc. ↑T-MAE ↓ST Err. ↓BERT-F1 ↑Latency (s) ↓
Qwen3-VL-RGB-4B78.134.460.51320.26662.34
Uniform + RGB-Text77.654.510.52950.26761.64
EGFS + DRTS78.204.500.51750.26981.66

The specific revisions are on "Benefits of Event Signals for Spatiotemporal Sparsification" on page 18 of the revised manuscript.


Q5: Dataset Release and Privacy Protection.

Data and Tool Release. We plan to publicly release the complete TRACE-Bench dataset upon publication, including synchronized RGB videos, raw event streams, and all associated annotations. In addition, we will release our annotation guidelines, behavior and interaction annotation protocols, and the source code of our annotation tool. By making both the dataset and its annotation infrastructure publicly available, we aim to facilitate reproducible evaluation, support further dataset extension, and promote the development of benchmarks for fine-grained traffic behavior and interaction understanding.

Licensing. We plan to release the dataset and annotation documentation under the CC BY-NC 4.0 license, and the annotation software under the MIT License.

Privacy Protection. To protect personal privacy, we have applied blurring to identifiable faces and vehicle license plates in the videos.


We have uploaded the revised manuscript, incorporating the additional results and clarifications discussed above. We welcome any further questions or suggestions and would be happy to provide additional clarification. We look forward to your feedback.

Thanks for the agreement numbers, the bootstrap CIs and the matched-budget RGB-only run, these cover most of my concerns so I raise my score to 3.

We thank the reviewer for their positive feedback and for reconsidering the score. We appreciate the reviewer's constructive comments, which have helped us strengthen the experimental validation and improve the rigor of our work.

Comments on the paper

3 comments

We have provided point-by-point responses to the reviewers' comments and uploaded the revised manuscript, with all changes highlighted in blue. We sincerely appreciate the reviewers' valuable feedback and look forward to further comments and discussion.

Dear Reviewers,

As the discussion period is approaching its end, we would like to kindly follow up on our responses and revised manuscript. We hope our additional experiments and clarifications have adequately addressed your concerns.

If you have any further questions or suggestions, we would be happy to address them. We sincerely appreciate your time and constructive feedback and look forward to hearing from you.

Best regards, The Authors

AuthorsOpenReview ↗

Summary of the discussion

We sincerely thank the Area Chair and all reviewers for their thoughtful evaluations and constructive feedback. We have carefully addressed the concerns through additional experiments and detailed clarifications. We would like to summarize the key contributions and evidence for the final assessment.

1. Distinct Contributions of TRACE-Bench

TRACE-Bench addresses an important gap between agent-level perception and holistic event reasoning by explicitly modeling continuous behavior evolution and directed agent interactions. Its evidence-grounded QA pairs link scene context, agent identities, temporal behaviors, and interactions within a hierarchical evaluation framework. Beyond aggregate accuracy, this structure enables cross-level error diagnosis, revealing where MLLMs fail as reasoning progresses from perception to behavior and interaction understanding. We believe this evidence-linked diagnostic capability is the central contribution of TRACE-Bench.

2. Additional Evidence Addressing the Reviewers' Concerns

  • Annotation Reliability and Reasoning Validity. We conducted a three-annotator agreement analysis, clarified the criteria for identifying directed interactions based on observable evidence, and independently verified the uniqueness of counterfactual answers under the specified hypotheses and candidate options.

  • Benchmark Evaluation Validity. We expanded the evaluation to include additional representative video MLLMs and conducted further analyses of language bias, template dependence, and answer parsing, strengthening the reliability and fairness of the benchmark's evaluation protocol.

  • Fair Method Comparisons and Inference Efficiency. We conducted matched-computational-budget comparisons and additional sparsification experiments using alternative motion cues, while clarifying the sampling strategies and efficiency baselines. These results further support the effectiveness of event-guided sparsification and its better accuracy–efficiency trade-off.

  • Statistical Reliability of Experimental Results. We conducted clip-level bootstrap analysis and hyperparameter sensitivity experiments, providing statistical support for the observed performance improvements and demonstrating the method's robustness across different configurations.

  • Dataset Composition and Split Reliability. We provided additional statistics on scenario distributions and clarified the camera-site-disjoint splitting protocol.

3. Closing Remarks

We recognize that broader environmental coverage would further strengthen the benchmark. Nevertheless, TRACE-Bench's primary value lies in its fine-grained, evidence-linked evaluation of behavior evolution and directed interaction reasoning, rather than dataset scale alone. The additional evidence strengthens the reliability of its annotations, the fairness of its evaluation, and the empirical support for the proposed framework.

We sincerely appreciate the Area Chair's and reviewers' time and valuable suggestions, and hope these contributions and additional findings will receive careful consideration in the final assessment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.