- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces EventBench, a benchmark for event-based MLLMs covering eight tasks across understanding, recognition, and spatial reasoning. It also constructs EQA-1.4M, a large-scale event–text instruction dataset. Experiments on proprietary, open-source, and event-based MLLMs demonstrate the benefits of event-specific training while revealing limitations in fine-grained recognition and spatial reasoning.
Critical strengths
-
EventBench unifies eight tasks across three evaluation dimensions, particularly introducing spatial reasoning tasks that are largely overlooked by existing event-based benchmarks.
-
The EQA-1.4M dataset and evaluations across multiple MLLM families provide valuable resources and empirical insights for event-based multimodal learning.
Critical weaknesses
-
Video-based MLLMs use event videos sampled at 1 FPS, whereas event-based models process raw event streams. This discrepancy may favor event-based models and weaken the conclusions about their superiority.
-
The benchmark relies partly on GPT-generated questions and publicly available video datasets. The paper provides limited details on annotation reliability and potential overlap with models' pretraining data.
-
Since 64.5% of the data is synthetic, the benchmark's ability to reflect performance on real event-camera streams remains unclear. Separate evaluations on real and synthetic data are missing.
-
The test set contains only 4,001 QA pairs, with some tasks having fewer than 200 samples. Statistical uncertainty is not reported, and the 30% distance-error tolerance lacks justification.
Questions for authors
-
How are annotation quality and potential training–test contamination verified, particularly for questions generated by GPT-5 and samples sourced from public video datasets?
-
Can the authors compare raw-event and event-video inputs under matched training settings and evaluate whether higher video sampling rates reduce the observed performance gap?
-
Can the authors report separate results on real and synthetic event streams to demonstrate the benchmark's real-world relevance?
6 comments
Response to Reviewer z8P4 (II/IV)
W2: Annotation reliability and potential pretraining overlap
Annotation reliability. To construct QA pairs suited for MLLMs across AR, GR, and E-OCR, we repurpose existing labels from real-event datasets (lines 183–184), which account for 35.5% of all test samples. Spatial answers are derived directly from instance annotations and 3D geometry (lines 185–190). For DU/CR samples, we prompt GPT with paired RGB and event data alongside available annotations, followed by manual verification of every generated sample (lines 181–183). Additionally, we apply a two-pass filtering pipeline, including a classifier trained on 10,000 manually labeled samples, to regulate motion content and ensure overall stream quality (lines 156–159).
Data separation. We separate EQA-1.4M and EventBench at the sample level, with additional scene-level separation for spatial tasks (lines 159–161). While exposure to public source videos during base-model pretraining cannot be fully ruled out given undisclosed pretraining data, we ensure that no EventBench test samples are used in the EQA-1.4M fine-tuning of event-based models. We will clarify this distinction in the revision.
Response to Reviewer z8P4 (III/IV)
W3: Performance on real and synthetic event data
Why include simulated events. Few existing real event-camera datasets fit MLLM evaluation well, and they tend to cover only a narrow range of scenes. To address this, we include events generated via v2e to expand our scene and task diversity. This lets us run more thorough evaluations, especially for spatial reasoning tasks built on metric 3D annotations. Simulated events make up 64.5% of our QA pairs (DU, CR, SR, AD, OC), and real event recordings account for the remaining 35.5% (AR, GR, E-OCR).
Separate results. The table below reorganizes the original Table 2 results based on event source. On the real-event recognition subset, EventGPT+(L) reaches 65.2%, while the best video-based baseline, InternVL-3.5, gets 51.1%. This shows our performance advantage still holds for tasks using physically captured event streams.
| Model | Real events: (AR / GR / E-OCR) | Synthetic events: (DU / CR / SR / AD / OC) |
|---|---|---|
| GPT-5 | 39.3 | 48.5 |
| GPT-4o | 41.0 | 42.6 |
| Gemini-2.5-Pro | 37.7 | 40.0 |
| Gemini-2.5-Flash | 35.7 | 34.3 |
| Doubao-Seed-1.6-Vision | 45.7 | 41.3 |
| Qwen2.5-VL | 44.6 | 52.4 |
| InternVL-3.5 | 51.1 | 59.3 |
| MiniCPM-V-4 | 46.8 | 55.9 |
| MiMo-VL | 42.5 | 58.2 |
| EventGPT | 62.7 | 64.0 |
| EventGPT+(B) | 63.0 | 64.3 |
| EventGPT+(L) | 65.2 | 65.7 |
Response to Reviewer z8P4 (I/IV)
We sincerely thank the reviewer for taking the time to carefully evaluate our work and provide detailed feedback. To our knowledge, EventBench is the first fully open benchmark for event-based MLLMs, provides openly released data, baseline models, evaluation code, and a public Codabench[3] leaderboard for event-based MLLMs. Its over 10,000 downloads and over 100 team registrations across two challenge tracks demonstrate community uptake and support its practical utility as a shared evaluation resource.
W1: Input representations and temporal sampling
Sampling robustness. All open-source models are trained on the EQA-1.4M split under identical SFT settings. As shown in Table, raising the event-video sampling rate from 1 FPS to 2 FPS, while keeping everything else unchanged, boosts Qwen2.5-VL by 0.2 points and InternVL-3.5 by 0.3 points only. At Δt=0.5 s, EventGPT+(L) still holds an 8.4-point lead over our best video baseline. This ranking stays consistent across all four sampling intervals. Our goal is to provide open benchmarking infrastructure for the event-based MLLM community, rather than to train a state-of-the-art model.
Performance under different sampling intervals Δt (in seconds).
Why 1 FPS. We set 1 FPS as our default for practicality, following existing conventions like the Qwen2.5-VL video inference example in the official Transformers docs [1]. This choice balances accuracy and inference latency. Higher sampling rates add extra frame computation, and Table 4 confirms they yield only small accuracy improvements. We will clarify the sampling units and make it clear our comparisons are run on full systems under the stated input protocols.