- Rating
- 2
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces TRACE-Bench, a dataset for testing whether VLMs can understand roadside traffic videos beyond simply detecting cars and pedestrians. It evaluates whether models can follow how road users move, interact, and reason about possible outcomes. The authors also propose a method that uses event-camera signals to select important video frames and regions, reducing unnecessary computation while improving traffic understanding. Experiments show that vision-language models still struggle with complex interactions.
Critical strengths
- The reviewer really likes the high quality presentation of the paper. Great work authors in putting together the manuscript and writing (of course with some need for improvements).
- The strongest part for the reviewer is TRACE-Bench’s explicit reasoning chain from perception → behavior → directed interaction, with each QA tied back to structured evidence rather than only a final answer. This diagnostic capability is much more useful for a benchmark than a simple overall accuracy metric. Evaluating how well responses are grounded reveals far more about whether and how these VLMs hallucinate.
Critical weaknesses
- So one thing most critical to the reviewer is that the authors label directed interactions as one agent inducing another agent’s response, BUT the dataset is observational roadside video. That does not establish causality: both agents could be responding for example to a traffic light, road geometry, another vehicle, or some other latent factor. The counterfactual task has the same problem: there is no true observed ground truth for what would have happened under the hypothetical change. In the reviewer’s opinion, this would hamper the “causal reasoning” claim of the paper.
- The effective dataset is also small: 201 videos. The headline “22,807 QA pairs” sounds large, but they come from only 201 sequences and roughly 1.4K agents. Moreover, questions are populated from manually designed templates. That means many examples are correlated and the effective visual diversity is much smaller than 22.8K suggests. This becomes particularly problematic when small performance gains are reported without video-level confidence intervals.
- Authors argue that event data helps identify informative frames and regions, but it does not compare against strong RGB-only motion cues such as optical flow, frame differences, background subtraction, etc. What’s the benefit from event sensing specifically, rather than simply from having any motion-based importance signal?
Questions for authors
- In Table 8-9, the proposed model gets 83.05% accuracy at night vs 71.96% during daytime. The same pattern appears for the RGB and Event-only baselines too. That is interesting because the paper motivates event cameras partly as being especially useful under difficult illumination. But these numbers suggest the night subset may simply be easier overall, so the day/night comparison is confounded by dataset difficulty. What do the authors think of this analysis?
- Qwen family models (2.5+) have used dynamic sampling for video processing. Why not use that, and come up with your own sampling method? Any comparisons?
5 comments
Part2
W3: Event-Specific Benefits over RGB-Based Motion Cues.
To assess the benefits of event signals over conventional RGB-based motion cues, we conduct additional ablations by replacing the event signals in our sparsification method with RGB frame differences and optical flow estimated by RAFT, while keeping all other settings unchanged.
As shown in the table below, the event-based method achieves 78.77 accuracy, outperforming frame differences and optical flow by 2.12 and 2.19, respectively, while yielding the lowest spatiotemporal localization error. Its inference latency is comparable to that of frame differences and substantially lower than that of RAFT, which introduces additional computational overhead for optical flow estimation. These results demonstrate that the high temporal resolution of event sensing enables finer-grained motion representation, facilitating more effective selection of motion-salient frames and regions.
| Method | Acc. ↑ | T-MAE ↓ | ST Err. ↓ | BERT-F1 ↑ | Latency (s) ↓ |
|---|---|---|---|---|---|
| Frame Difference | 76.65 | 4.27 | 0.5198 | 0.2741 | 1.79 |
| Optical Flow (RAFT) | 76.58 | 4.25 | 0.5125 | 0.2697 | 2.45 |
| Event (Ours) | 78.77 | 4.25 | 0.5009 | 0.2752 | 1.82 |
The specific revisions are on "Benefits of Event Signals for Spatiotemporal Sparsification" on page 18 of the revised manuscript.
Q1: Day–Night Performance Discrepancy and Event Camera Advantages under Low-Light Conditions.
We thank the reviewer for this insightful analysis. First, higher nighttime accuracy does not necessarily indicate that the nighttime subset is easier across all tasks. As shown in Tables 8–9, although our method achieves higher accuracy at night, its spatial and spatiotemporal localization errors increase (S-L1: 0.1390 vs. 0.1133; ST Err.: 0.5402 vs. 0.4217), while BERT-F1 for open-ended QA decreases (0.2449 vs. 0.3047). Similar trends are observed for the RGB-only and Event-only baselines. These results suggest that day–night performance differences vary across tasks and may reflect both illumination conditions and differences in scene difficulty between the two subsets.
Nevertheless, comparisons within the same evaluation subsets provide evidence of the potential complementary value of event data under low-light conditions. Specifically, on the daytime subset, RGB-only outperforms Event-only in BERT-F1 and BLEU-4. However, on the nighttime subset, Event-only surpasses RGB-only on both metrics (0.2457 vs. 0.2404 and 0.0466 vs. 0.0431). This shift in relative performance suggests that event data can provide potential complementary information under low-light conditions. We will clarify the interpretation and scope of the day–night evaluation in the revised manuscript.
The specific revisions are on "A.2.2 Performance Under Different Illumination Conditions" on page 16 of the revised manuscript.
Q2: Event-Guided Sampling vs. Qwen's Dynamic Video Sampling.
We would like to clarify that Qwen's native dynamic sampling mechanism selects frames at approximately uniform temporal intervals according to the different FPS or frame budget. Therefore, it does not explicitly prioritize frames based on their motion content. In contrast, our Event-Guided Frame Sampling (EGFS) leverages event activity and temporal novelty to identify dynamically informative moments, while retaining uniformly sampled frames to preserve global temporal context. Furthermore, in Section 5.3 (Temporal Sampling Strategy), we directly compare EGFS with the uniform temporal sampling strategy used in Qwen under the same frame budget. The results show that EGFS achieves higher accuracy, demonstrating the effectiveness of event-guided selection in preserving informative temporal cues for traffic understanding.
We have uploaded the revised manuscript, incorporating the additional results and clarifications discussed above. We welcome any further questions or suggestions and would be happy to provide additional clarification. We look forward to your feedback.
Response to rebuttal
Hi Authors,
First of all, thank you for your hard work.
-
Given your response to W3, the reviewer would always choose "Frame Difference" as it is easy to implement and scale. The reviewer doesn't see the value in Event (Ours) for the spatiotemporal sparsification based the triviality of the gains.
-
For the Qwen-style sampling, the reviewer thinks it is beyond "simple" uniform sampling. "Dynamic" and "uniform" won't go hand in hand, right? Even if conceding that "uniform" is approximating the "dynamic" sampling, the gains are again trivial (Table 5).
-
The above two points brings the reviewer to the main point which other reviewers are raising as well: The reviewer believes the work is being limited by the number of video samples in the dataset. In the response to Reviewer udWP's comment, the numbers do not present the scale of benchmark needed to make strong conclusions. This resonates from the reviewer's own experience as well.
Given the above points, the reviewer maintains the rating of "Weak Reject". The reviewer will not choose "Clear Reject" as the paper is still in the right direction. Hope the comments help!
Part1
We sincerely appreciate your highly positive assessment of the manuscript's presentation and the value of TRACE-Bench, as well as your insightful and constructive comments. We have carefully considered your concerns and provide our point-by-point responses below.
W1: Causal Validity of Directed Interactions and Counterfactual Reasoning.
For directed interaction annotation, we do not infer interactions solely from the temporal order of two agents' behaviors. Instead, we require sufficient visual evidence of an influence–response relationship, considering their relative spatial positions, behavioral changes, temporal order, and surrounding traffic context. An interaction is annotated only when these cues collectively support a plausible interpretation. Cases with insufficient visual evidence or where behavioral changes cannot be distinguished from responses to shared environmental factors are excluded. We use roadside videos because their wider field of view provides more comprehensive observations of traffic participants and their surroundings, supporting more informed interaction judgments.
Although annotating interactions in complex traffic scenes is challenging, understanding influence–response relationships is essential for analyzing traffic dynamics. We therefore believe that constructing such annotations under explicit evidence constraints is both meaningful and necessary. To ensure reliability, we establish consistent interaction annotation criteria and independently review interaction labels, focusing on whether the interacting agents, response behaviors, and spatiotemporal relationships are supported by observable evidence.
For counterfactual questions, the task evaluates whether a model can identify the potential consequence or safety-related mechanism most consistent with the hypothetical condition and available visual evidence. Answer uniqueness is defined within the given hypothesis and answer options, rather than implying a uniquely determined or verifiable real-world counterfactual outcome. All counterfactual questions undergo independent review, and cases with ambiguous hypotheses or multiple plausible answers are revised or excluded.
We have revised the manuscript to avoid equating evidence-grounded interaction understanding with causal inference, and clarified that the counterfactual task assesses plausibility under specified hypothetical conditions rather than estimating actual causal effects. The specific revisions are in "Abstract" on page 1, "Introduction" on pages 2 and 3, "A.3.3 QA Construction" and "A.3.4 Quality Control" on pages 20 and 21 of the revised manuscript.
W2: Visual Diversity, Correlated QA Samples, and Statistical Reliability.
The primary contribution of TRACE-Bench is not large-scale scenario coverage, but a fine-grained evaluation framework with explicit spatiotemporal evidence and cross-level associations, addressing the limitations of existing benchmarks in evaluating continuous behavior evolution and directed interaction understanding. This requires constructing interconnected QA pairs from agent, behavior, and interaction annotations within the same video, enabling systematic evaluation, as well as cross-level error attribution. Thus, the 22,807 QA pairs primarily reflect the granularity and scope of the evaluation tasks, offering research value distinct from simply increasing the number of videos or environmental categories.
The fine-grained annotations in TRACE-Bench require substantial human effort and rigorous verification, ensuring high-quality structured annotations while limiting dataset scale. We will clarify this limitation in the revised manuscript and explore more efficient annotation methods to expand dataset scale and scenario coverage in the future.
To assess the statistical reliability of the performance gains, we conducted a paired, clip-level bootstrap analysis on the validation set, using 10,000 resamples with replacement at the video-clip level. The resulting 95% confidence interval for the accuracy improvement over the RGB-only Qwen3-VL-4B baseline is [0.26, 1.44], providing statistical support for the observed gain. Furthermore, compared with the RGB-only baseline, our method reduces inference latency by 22.2% and FLOPs by 60.5%, demonstrating a better accuracy–efficiency trade-off.
The specific revisions are in "Limitations and Future Work" on page 10 and "Fine-tuned Performance" on page 9 of the revised manuscript.