acceptodds
Withdrawn from ICLR 2027

EventTimeQA: Auditing Intervention-Consistent Millisecond Temporal Representations

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Event-stream and high-frame-rate benchmarks usually score samples independently, which cannot show whether a model changes its prediction for the right reason when one temporal variable changes. We present EventTimeQA, a controlled synthetic evaluation of intervention-consistent millisecond temporal perception and representation. Its 10,000 factual/counterfactual pairs hold scene nuisance variables fixed while swapping onset order, relative frequency, or time-to-contact. We evaluate predictions with pair accuracy and flip consistency, temporal-resolution sweeps, timestamp perturbations, scene and sensor shifts, and an explicit audit of post-simulation event-retention policies. Frozen v3 evidence shows that lightweight supervised event representations approach ceiling performance, while visual-language performance depends strongly on interface: most tested event-contact-sheet interfaces remain weak, whereas a budget-compliant third-party Gemini route reaches pair accuracy on RGB500 but at most on event input. A leakage-audited 0/2/4/8-shot study further shows non-monotonic, model- and interface-specific effects. We release executable construction, shortcut, statistical, documentation, and maintenance protocols. The benchmark and all empirical claims are deliberately restricted to the controlled synthetic domain; physical-sensor transfer and human performance are outside the scope of this release.

Primary area:
datasets and benchmarks
Keywords:
event-based vision, temporal perception, counterfactual evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_L3SAOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper introduces EventTimeQA, a synthetic benchmark for millisecond-scale temporal understanding using event streams. Each factual/counterfactual pair changes one of three temporal variables—motion onset, flicker frequency, or time to contact—while keeping the remaining scene factors fixed. The paper evaluates supervised event encoders, pretrained event backbones, and several general-purpose VLMs under different temporal representations and retention policies. While the benchmark contains an unusually large amount of auditing and bookkeeping, I found the actual scientific contribution rather thin: the task is highly constrained, largely synthetic, and appears much closer to temporal binary classification than to meaningful event-based question answering.

Critical strengths
  1. The paired factual/counterfactual construction and the use of pair accuracy are sensible controls against constant-side predictions. The authors also perform a number of shortcut, retention-policy, timestamp, and resolution checks, although much of this feels like validation of the benchmark construction itself rather than a substantive scientific contribution.
  2. The paper evaluates several model families and reports a relatively large number of diagnostic settings. The authors are also reasonably explicit about some limitations, including the lack of physical-sensor and human evaluation.
Critical weaknesses

Before discussing the technical issues, I find the presentation quality surprisingly careless for a conference submission. Figure 1 strongly resembles an AI-generated or AI-assisted ideation graphic rather than a publication-ready scientific figure: it contains tiny and partially malformed text, is visibly rasterized/blurry, and even includes the phrase “Original reference-style drawing for scientific figure ideation.” The remaining figures are also rough; for example, Figure 2 draws a visual guide through unreported 8/16 ms measurements and labels the raw-RGB baseline only as “do poorly” without reporting the actual numbers. This is particularly concerning for a benchmark paper whose main value is supposed to be careful evaluation and reporting.

  1. The benchmark is too narrow and appears surprisingly easy to solve without the reasoning capability it claims to study. The tasks consist of only three procedurally generated binary temporal comparisons. More importantly, a simple temporal-histogram ridge classifier without spatial coordinates already reaches 92.81–97.01% pair accuracy, while one-sided temporal input reaches 82.0–86.8%, and a small supervised GRU reaches 99.33%. These results strongly suggest that paired counterfactual reasoning, spatial understanding, or compositional QA is not actually required. The paper itself eventually describes the task as “temporal-intervention classification,” which makes the broader EventTimeQA framing difficult to justify.

  2. The positioning against prior work and the experimental scope are inadequate for a benchmark paper. There is no proper Related Work section, and the brief “Task and Positioning” discussion does little more than list neighboring benchmarks. In particular, EventGPT and EventBench are mentioned but not meaningfully analyzed, and EventGPT—a directly relevant event-based MLLM—is not evaluated, while many generic VLMs are instead tested through an ad-hoc event contact-sheet interface. The benchmark is also entirely based on 64×64 procedural synthetic scenes, with no physical event-camera data or human evaluation, making its external validity very limited.

Questions for authors
  • If a temporal histogram without spatial coordinates reaches up to 97.01% pair accuracy, and one-sided temporal input already reaches 82.0–86.8%, what capability is EventTimeQA actually testing beyond simple temporal classification? Why is the paired QA formulation necessary?

  • Why is there no dedicated Related Work section for an explicitly benchmark-oriented paper? How does EventTimeQA substantially differ from EventGPT, EventBench, FPS-Bench, TemporalBench, and existing temporal-counterfactual benchmarks beyond combining several already existing ingredients?

  • Why is EventGPT not evaluated? It appears to be one of the most directly relevant event-based multimodal models, yet the experiments instead contain a large collection of generic VLMs evaluated through event contact sheets.

  • Was generative AI used to produce Figure 1? If so, why is this not stated in the AI-use disclosure? What is the origin of the phrase “Original reference-style drawing for scientific figure ideation” appearing directly inside the submitted figure? The current figure quality is not suitable for publication.

  • Why does Figure 2(c) visually connect measurements across 8 and 16 ms when those values are explicitly stated as unreported? Similarly, why are the raw-RGB results described only as “do poorly” instead of reporting the actual values?

  • What evidence supports external validity beyond the synthetic generators? Both the main v2e data and the “independent generator” remain simulated, and no physical event-camera or human evaluation is provided.

  • The paper highlights externally evaluated Gemini results in the abstract, yet later states that the upstream revision and direct-provider provenance are unverified. Why should such results be emphasized in the headline findings of a benchmark paper?

Reviewer_G8orOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

EventTimeQA tests whether models follow millisecond-scale temporal changes. Each pair of samples changes one temporal variable (motion onset, flicker frequency, or time to contact) and reverses the binary answer, while the rest of the synthetic scene stays fixed. A pair counts as correct only when both answers are correct. To remove shortcuts from coarse event counts, the released streams are thinned so that the two members of a pair have identical 32-ms event histograms, and only finer timing decides the answer. The paper evaluates supervised event encoders, frozen pretrained event backbones, and a range of VLMs given event contact sheets or RGB frames. It adds controls with alternative thinning rules and a second event simulator, and limits its conclusions to synthetic scenes.

Critical strengths
  1. Pair accuracy is a well-chosen metric for this question. A model that always picks the same side scores 50% sample accuracy but zero pair accuracy, so the metric exposes failures that sample accuracy hides. The paper explains this clearly, separates constant-side prediction from random guessing (25% pair accuracy), and computes confidence intervals over complete pairs, which matches how the data are constructed.

  2. The controls are carefully designed. Comparing raw, first-kept, and randomly kept event streams, shuffling timestamps, and repeating the test with an independently written simulator together make it unlikely that the supervised results come from a single construction artifact. The authors also report findings that limit their own claims, such as the strong one-sided classifiers and the 100% accuracy of RGB frame differences at 240 FPS.

Critical weaknesses
  1. The benchmark construction is not specified well enough to reproduce or check. The paper does not give the task-generation equations and parameter ranges, the length of the evidence window, the definition and size of the scene and sensor OOD splits, or the simulator configuration. These details determine what the intervention actually changes. The training recipes are also missing, and they are needed to explain why the same GRU stays clearly below ceiling in Section 3 but reaches ceiling in Table 2. The release records mentioned in the paper do not accompany the submission. In addition, Table 11 is described as covering 1,000 pairs per policy, yet its temporal-control scores and interval widths fit a test set of only about 167 pairs. Please state the actual test split and how these scores are computed.

  2. It is unclear how much temporal information each model receives, which makes the gap between interfaces hard to interpret. By construction, the two members of a pair become identical once events are summed over aligned 32-ms bins, which is why accuracy drops to 0% at 32 ms in Figure 2(c). Any input with bins of 32 ms or longer therefore cannot solve the task, regardless of the model. The paper does not report the window length or the bin boundaries, so it remains open whether the histogram bins used for RVT and TESPEC and the twelve slices of the event contact sheets are shorter than 32 ms. RGB500 raises the same question. Twelve consecutive frames at 500 FPS cover only about 22 ms, while twelve frames spread over a longer window correspond to a much lower frame rate. Since the abstract highlights the RGB-versus-event gap of VLMs, please give the frame timestamps and bin boundaries for each interface. The per-task and temporal-gap results announced in Section 4 and the values left unreported in Figure 2 would also show whether the low scores come from the input or from the models.

Questions for authors
  1. Which test split does Table 11 use, and which training settings produce the GRU results in Section 3 and Table 2?
  2. What window length, bin boundaries, and frame timestamps does each interface use, and do the per-task and temporal-gap results show that the temporal signal survives each model's preprocessing?
  3. Section 5 reports the best open-VLM score on event contact sheets as 13.0%, a value that does not appear in any table. Which result is meant? In addition, the Gemini scores in the abstract are the best over shot counts on the test set. Please mark them as such or select the shot count on a separate split.
Reviewer_2fHGOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces EventTimeQA, a synthetic benchmark of 10,000 factual–counterfactual pairs for testing millisecond-scale temporal perception in event-based and multimodal models. Each pair changes one temporal variable (onset order, flicker frequency, or time-to-contact) while holding everything else fixed, so the evaluation can check whether models respond correctly to interventions rather than only measuring accuracy. Lightweight supervised models reach near-ceiling performance, while pretrained models and vision-language models vary widely across representations and interfaces. The results show that fine temporal resolution matters and that few-shot prompting has non-monotonic effects. The authors limit their conclusions to controlled synthetic settings.

Critical strengths

Comprehensive evaluation with rigorous shortcut and robustness analysis: The paper validates the benchmark extensively through temporal-resolution experiments, timestamp perturbations, scene and sensor distribution shifts, alternative event-retention policies, and an independently implemented event generator. Shortcut controls and pair-level statistical analysis give confidence that the benchmark measures genuine temporal information rather than nuisance correlations. The experiments also expose clear differences between supervised event representations, pretrained models, and vision-language interfaces, which offers useful insight into the limitations of current temporal perception systems.

Critical weaknesses

Limited benchmark difficulty and discriminative power. Although the controlled counterfactual design is valuable, the benchmark seems relatively easy for models trained on its data. Lightweight supervised event encoders reach near-ceiling pair accuracy (98–99%) in distribution and stay similarly high under scene shifts. Even one-sided temporal inputs reach 82.0–86.8% pair accuracy, which suggests that strong performance does not require sophisticated relational or compositional temporal reasoning. This raises doubts about whether the benchmark can distinguish stronger temporal representations or remain useful as models improve, which may limit its long-term impact.

Questions for authors

Could the authors clarify which temporal capabilities the paired counterfactual evaluation measures beyond those that simple temporal classifiers already capture? This would help establish the benchmark’s long-term value.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.