- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper introduces EventTimeQA, a synthetic benchmark for millisecond-scale temporal understanding using event streams. Each factual/counterfactual pair changes one of three temporal variables—motion onset, flicker frequency, or time to contact—while keeping the remaining scene factors fixed. The paper evaluates supervised event encoders, pretrained event backbones, and several general-purpose VLMs under different temporal representations and retention policies. While the benchmark contains an unusually large amount of auditing and bookkeeping, I found the actual scientific contribution rather thin: the task is highly constrained, largely synthetic, and appears much closer to temporal binary classification than to meaningful event-based question answering.
Critical strengths
- The paired factual/counterfactual construction and the use of pair accuracy are sensible controls against constant-side predictions. The authors also perform a number of shortcut, retention-policy, timestamp, and resolution checks, although much of this feels like validation of the benchmark construction itself rather than a substantive scientific contribution.
- The paper evaluates several model families and reports a relatively large number of diagnostic settings. The authors are also reasonably explicit about some limitations, including the lack of physical-sensor and human evaluation.
Critical weaknesses
Before discussing the technical issues, I find the presentation quality surprisingly careless for a conference submission. Figure 1 strongly resembles an AI-generated or AI-assisted ideation graphic rather than a publication-ready scientific figure: it contains tiny and partially malformed text, is visibly rasterized/blurry, and even includes the phrase “Original reference-style drawing for scientific figure ideation.” The remaining figures are also rough; for example, Figure 2 draws a visual guide through unreported 8/16 ms measurements and labels the raw-RGB baseline only as “do poorly” without reporting the actual numbers. This is particularly concerning for a benchmark paper whose main value is supposed to be careful evaluation and reporting.
-
The benchmark is too narrow and appears surprisingly easy to solve without the reasoning capability it claims to study. The tasks consist of only three procedurally generated binary temporal comparisons. More importantly, a simple temporal-histogram ridge classifier without spatial coordinates already reaches 92.81–97.01% pair accuracy, while one-sided temporal input reaches 82.0–86.8%, and a small supervised GRU reaches 99.33%. These results strongly suggest that paired counterfactual reasoning, spatial understanding, or compositional QA is not actually required. The paper itself eventually describes the task as “temporal-intervention classification,” which makes the broader EventTimeQA framing difficult to justify.
-
The positioning against prior work and the experimental scope are inadequate for a benchmark paper. There is no proper Related Work section, and the brief “Task and Positioning” discussion does little more than list neighboring benchmarks. In particular, EventGPT and EventBench are mentioned but not meaningfully analyzed, and EventGPT—a directly relevant event-based MLLM—is not evaluated, while many generic VLMs are instead tested through an ad-hoc event contact-sheet interface. The benchmark is also entirely based on 64×64 procedural synthetic scenes, with no physical event-camera data or human evaluation, making its external validity very limited.
Questions for authors
-
If a temporal histogram without spatial coordinates reaches up to 97.01% pair accuracy, and one-sided temporal input already reaches 82.0–86.8%, what capability is EventTimeQA actually testing beyond simple temporal classification? Why is the paired QA formulation necessary?
-
Why is there no dedicated Related Work section for an explicitly benchmark-oriented paper? How does EventTimeQA substantially differ from EventGPT, EventBench, FPS-Bench, TemporalBench, and existing temporal-counterfactual benchmarks beyond combining several already existing ingredients?
-
Why is EventGPT not evaluated? It appears to be one of the most directly relevant event-based multimodal models, yet the experiments instead contain a large collection of generic VLMs evaluated through event contact sheets.
-
Was generative AI used to produce Figure 1? If so, why is this not stated in the AI-use disclosure? What is the origin of the phrase “Original reference-style drawing for scientific figure ideation” appearing directly inside the submitted figure? The current figure quality is not suitable for publication.
-
Why does Figure 2(c) visually connect measurements across 8 and 16 ms when those values are explicitly stated as unreported? Similarly, why are the raw-RGB results described only as “do poorly” instead of reporting the actual values?
-
What evidence supports external validity beyond the synthetic generators? Both the main v2e data and the “independent generator” remain simulated, and no physical event-camera or human evaluation is provided.
-
The paper highlights externally evaluated Gemini results in the abstract, yet later states that the upstream revision and direct-provider provenance are unverified. Why should such results be emphasized in the headline findings of a benchmark paper?