acceptodds
Under review as a conference paper at ICLR 2027

EventBench: Towards Comprehensive Benchmarking of Event-based MLLMs

Abstract

Multimodal large language models (MLLMs) have made significant advancements in event-based vision. However, the comprehensive evaluation of these models' capabilities within a unified benchmark remains a crucial yet largely unexplored aspect. In this paper, we introduce **EventBench**, which presents an evaluation benchmark covering 8 diverse task metrics and a large-scale event stream dataset. Our work distinguishes existing event-based benchmarks in four key aspects: **i) Openness in accessibility**, releasing all raw event streams and task instructions across 8 evaluation metrics; **ii) Diversity in task coverage**, covering diverse understanding, recognition, and spatial reasoning tasks, enabling comprehensive evaluation of model capability; **iii) Integration in spatial dimensions**, pioneering the design of 3D spatial reasoning task for event-based MLLMs; **iv) Scale in data volume**, with an accompanying training set of over one million event–text pairs supporting large-scale training and evaluation. With EventBench, we evaluate state-of-the-art closed-source models such as GPT-5 and Gemini-2.5 Pro, leading open-source models including Qwen2.5-VL and InternVL3, as well as event-based MLLMs like EventGPT that directly process raw event inputs. Extensive evaluation reveals that current event-based MLLMs perform well in event stream understanding, yet they still struggle with fine-grained recognition and spatial reasoning.

Primary area:
datasets and benchmarks
Keywords:
Event-based Vision

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_z8P4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces EventBench, a benchmark for event-based MLLMs covering eight tasks across understanding, recognition, and spatial reasoning. It also constructs EQA-1.4M, a large-scale event–text instruction dataset. Experiments on proprietary, open-source, and event-based MLLMs demonstrate the benefits of event-specific training while revealing limitations in fine-grained recognition and spatial reasoning.

Critical strengths
  1. EventBench unifies eight tasks across three evaluation dimensions, particularly introducing spatial reasoning tasks that are largely overlooked by existing event-based benchmarks.

  2. The EQA-1.4M dataset and evaluations across multiple MLLM families provide valuable resources and empirical insights for event-based multimodal learning.

Critical weaknesses
  1. Video-based MLLMs use event videos sampled at 1 FPS, whereas event-based models process raw event streams. This discrepancy may favor event-based models and weaken the conclusions about their superiority.

  2. The benchmark relies partly on GPT-generated questions and publicly available video datasets. The paper provides limited details on annotation reliability and potential overlap with models' pretraining data.

  3. Since 64.5% of the data is synthetic, the benchmark's ability to reflect performance on real event-camera streams remains unclear. Separate evaluations on real and synthetic data are missing.

  4. The test set contains only 4,001 QA pairs, with some tasks having fewer than 200 samples. Statistical uncertainty is not reported, and the 30% distance-error tolerance lacks justification.

Questions for authors
  1. How are annotation quality and potential training–test contamination verified, particularly for questions generated by GPT-5 and samples sourced from public video datasets?

  2. Can the authors compare raw-event and event-video inputs under matched training settings and evaluate whether higher video sampling rates reduce the observed performance gap?

  3. Can the authors report separate results on real and synthetic event streams to demonstrate the benchmark's real-world relevance?

6 comments
AuthorsOpenReview ↗

Response to Reviewer z8P4 (I/IV)

We sincerely thank the reviewer for taking the time to carefully evaluate our work and provide detailed feedback. To our knowledge, EventBench is the first fully open benchmark for event-based MLLMs, provides openly released data, baseline models, evaluation code, and a public Codabench[3] leaderboard for event-based MLLMs. Its over 10,000 downloads and over 100 team registrations across two challenge tracks demonstrate community uptake and support its practical utility as a shared evaluation resource.

W1: Input representations and temporal sampling

Sampling robustness. All open-source models are trained on the EQA-1.4M split under identical SFT settings. As shown in Table, raising the event-video sampling rate from 1 FPS to 2 FPS, while keeping everything else unchanged, boosts Qwen2.5-VL by 0.2 points and InternVL-3.5 by 0.3 points only. At Δt=0.5\Delta t=0.5 s, EventGPT+(L) still holds an 8.4-point lead over our best video baseline. This ranking stays consistent across all four sampling intervals. Our goal is to provide open benchmarking infrastructure for the event-based MLLM community, rather than to train a state-of-the-art model.

Performance under different sampling intervals Δt\Delta t (in seconds).

ModelParams0.5 s1.0 s (default)1.5 s2.0 s
Qwen2.5-VL3B49.749.548.948.7
InternVL-3.54B56.556.255.955.6
EventGPT7B62.663.562.462.2
EventGPT+(L)7B64.965.564.363.9

Why 1 FPS. We set 1 FPS as our default for practicality, following existing conventions like the Qwen2.5-VL video inference example in the official Transformers docs [1]. This choice balances accuracy and inference latency. Higher sampling rates add extra frame computation, and Table 4 confirms they yield only small accuracy improvements. We will clarify the sampling units and make it clear our comparisons are run on full systems under the stated input protocols.

AuthorsOpenReview ↗

Response to Reviewer z8P4 (II/IV)

W2: Annotation reliability and potential pretraining overlap

Annotation reliability. To construct QA pairs suited for MLLMs across AR, GR, and E-OCR, we repurpose existing labels from real-event datasets (lines 183–184), which account for 35.5% of all test samples. Spatial answers are derived directly from instance annotations and 3D geometry (lines 185–190). For DU/CR samples, we prompt GPT with paired RGB and event data alongside available annotations, followed by manual verification of every generated sample (lines 181–183). Additionally, we apply a two-pass filtering pipeline, including a classifier trained on 10,000 manually labeled samples, to regulate motion content and ensure overall stream quality (lines 156–159).

Data separation. We separate EQA-1.4M and EventBench at the sample level, with additional scene-level separation for spatial tasks (lines 159–161). While exposure to public source videos during base-model pretraining cannot be fully ruled out given undisclosed pretraining data, we ensure that no EventBench test samples are used in the EQA-1.4M fine-tuning of event-based models. We will clarify this distinction in the revision.

AuthorsOpenReview ↗

Response to Reviewer z8P4 (III/IV)

W3: Performance on real and synthetic event data

Why include simulated events. Few existing real event-camera datasets fit MLLM evaluation well, and they tend to cover only a narrow range of scenes. To address this, we include events generated via v2e to expand our scene and task diversity. This lets us run more thorough evaluations, especially for spatial reasoning tasks built on metric 3D annotations. Simulated events make up 64.5% of our QA pairs (DU, CR, SR, AD, OC), and real event recordings account for the remaining 35.5% (AR, GR, E-OCR).

Separate results. The table below reorganizes the original Table 2 results based on event source. On the real-event recognition subset, EventGPT+(L) reaches 65.2%, while the best video-based baseline, InternVL-3.5, gets 51.1%. This shows our performance advantage still holds for tasks using physically captured event streams.

ModelReal events: (AR / GR / E-OCR)Synthetic events: (DU / CR / SR / AD / OC)
GPT-539.348.5
GPT-4o41.042.6
Gemini-2.5-Pro37.740.0
Gemini-2.5-Flash35.734.3
Doubao-Seed-1.6-Vision45.741.3
Qwen2.5-VL44.652.4
InternVL-3.551.159.3
MiniCPM-V-446.855.9
MiMo-VL42.558.2
EventGPT62.764.0
EventGPT+(B)63.064.3
EventGPT+(L)65.265.7
Reviewer_9vFiOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
2(+1)
Presentation
2
Soundness
2(+1)
Read review
Paper summary

This paper introduces a new benchmark for MLLM evaluation based on event camera. The proposed benchmark, EventBench, is established upon several existing event-based datasets, covering diverse tasks including understanding, recognition and spatial reasoning tasks. Then, these over one million event-text pairs and 4 k QA pairs are evaluated through some open-source and close-source MLLMs. The results show that existing MLLMs still struggle with fine-grained recognition and spatial reasoning.

Critical strengths
  1. The community needs a unified benchmark to evaluate event-based MLLMs across different abilities.

  2. The spatial tasks, including distance estimation, spatial relationships and object counting, could be useful for evaluating and improving the reasoning ability of MLLMs.

Critical weaknesses
  1. There are several inconsistencies between the text, figures and tables. For example, Fig. 3 reports 162 Event OCR questions out of 4,001 questions, which is about 4.05%, while Fig. 4 reports 11.6% for text matching. There are also repeated paragraphs. On page 4, lines 191–195 repeat the caption of Fig. 3 at lines 174–178. The two analysis paragraphs at lines 416–429 and 430–439, across pages 8–9, also repeat almost the same content. These problems make the paper difficult to follow and raise concerns about whether the reported statistics are consistent.

  2. The benchmark mainly combines existing datasets and converts their annotations into QA tasks. The contribution beyond this combination is not clearly demonstrated. Some important experimental details are also unclear, such as how spatial ground-truth answers are constructed. Moreover, the models use different input representations and training settings, so the results do not clearly show where the performance gains come from.

Questions for authors
  1. Please explain the inconsistencies above and the repeated paragraphs, and revise them.
  2. What new evaluation ability does EventBench provide compared with using existing benchmarks separately?
  3. How are the ground-truth answers for spatial tasks generated?
6 comments
AuthorsOpenReview ↗

Response to Reviewer 9vFi (I/II)

We sincerely thank the reviewer for taking the time to carefully evaluate our work and provide detailed feedback. To our knowledge, EventBench is the first fully open benchmark for event-based MLLMs. It includes the full dataset, baselines, evaluation code, and a leaderboard hosted on Codabench [4]. It has reached over 10,000 downloads and hosted a public challenge with 100+ teams across general understanding and spatial reasoning tracks, with top-three winners for each. This community adoption confirms its value as a practical tool for reproducible evaluation.

W1: Consistency of dataset statistics and repeated text

Statistics. Thanks for pointing out this confusion. Figure 3 breaks down results by task category, while Figure 4(c) groups them by answer type. The text-matching bucket combines E-OCR (162) and SR (301) samples, giving (162 + 301) / 4,003 ≈ 11.6%. SR entries follow the VSI-100k setup and use relation-based text answers [1]. We will spell out this difference in the revision so readers can readily tell them apart. As an open benchmark, EventBench’s data and annotations are already available on Hugging Face via the stated access process, so other researchers can reproduce and check our results independently.

Repeated text. We apologize for the oversight. The duplicated caption in lines 191–195 is an uncommented LaTeX artifact and will be removed. The discussions in lines 416–429 and 430–439 address two distinct questions: model capability bottlenecks (Table 2) and performance improvements brought by instruction tuning (Table 3). We will revise the text to reduce redundancy and better distinguish these two analyses.

AuthorsOpenReview ↗

Response to Reviewer 9vFi (II/II)

W2: Contribution, spatial ground truth, and experimental comparability

Our contribution. EventBench offers fully open evaluation infrastructure to develop and compare event-based MLLMs across eight tasks, spanning general understanding, reasoning, and spatial reasoning. Our baseline models supply reference performance numbers and a functional end-to-end evaluation pipeline. We emphasize that our core contribution is not a new state-of-the-art model.

To our knowledge, EventBench is the first benchmark to include spatial reasoning in event-based MLLM evaluation, with the event streams, text instructions, and baseline scores fully released.

We built a data and evaluation pipeline that applies multi-stage filtering, generates task-specific instructions, defines QA and scoring protocols, and uses v2e [2] to broaden scene and task coverage. We release the source data, event streams, QA pairs, models, and evaluation code, and host the evaluation on Codabench [4], an open-source platform for benchmarks and competitions.

Spatial ground truth. For the ScanNet-v2 subset, each instance ID identifies an object's semantic category and its mesh segments [3]. We use these annotations and the reconstructed geometry to compute the labels:

  • Object Counting (OC): We count unique instance IDs of the queried category across the entire reconstructed scene. Each object is counted once, including when it appears in multiple video observations.
  • Absolute Distance (AD): We compute the Euclidean distance between the two instances' 3D centers, d(A,B)=∥cA−cB∥2d(A,B)=\|c_A-c_B\|_2, in meters.
  • Spatial Relationship (SR): The observer stands at A and faces B; the question asks where C lies relative to this viewpoint. The A-to-B direction and the scene's upright axis define the reference frame used to assign C's textual relation label.

Input protocol. We evaluate each system using its native input interface: event-based models take raw event streams, while video MLLMs work on same source recordings. We stick to each model’s official setup. All open models use the EQA-1.4M split and matching SFT configurations, and every model runs on the identical test questions with the same scoring criteria.

Input sensitivity. We vary the sampling interval from 0.5 to 2 s (2 to 0.5 FPS for video inputs; Table 4) and the aggregation window from 10 to 40 ms (Table 5). The model ordering is consistent across the tested settings.

SettingQwen2.5-VLInternVL-3.5EventGPT+(L)
Sampling interval: 0.5 s (Table 4)49.756.564.9
Sampling interval: 2.0 s (Table 4)48.755.663.9
Aggregation window: 10 ms (Table 5)49.255.664.8
Aggregation window: 40 ms (Table 5)48.254.363.9

References

[1] Liao et al. Improved Visual-Spatial Reasoning via R1-Zero-Like Training. 2025.

[2] Hu et al. v2e: From Video Frames to Realistic DVS Events. CVPR Workshops, 2021.

[3] Dai et al. ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes. CVPR, 2017.

[4] Xu et al. Codabench: Flexible, easy-to-use, and reproducible meta-benchmark platform.

AuthorsOpenReview ↗

A Kind Reminder

We would like to kindly follow up on our rebuttal, as the discussion period is approaching its end. We hope our additional experiments and clarifications have addressed your concerns. We would greatly appreciate any further feedback you may have before the discussion closes. Thank you again for your time and consideration.

Reviewer_ymcpOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper describes EventBench, which is a public benchmark for benchmarking multimodal LLMs that use event-based cameras. Areas of focus are understanding, spatial reasoning and recognition. The paper compares the benchmark results across a range of close and open source MLLMs.

Critical strengths

The primary reason to accept is the establishment of a reasonable benchmark that helps to bring together the research community around the study of event-camera based MLLMs.

Critical weaknesses

The model set is broad, but the comparison is not like for like: frame models see 1 FPS renderings, closed models are not fine-tuned, event models get raw streams, and the winner is the authors' own baseline. The missing baseline is any RGB model on the paired video. GPT-5 helped write the understanding questions and is then scored on them, which is a possible bias even with human checking. Spatial Relationship has only 2–3 options, so its best score of 53% is close to chance. There are no seeds or confidence intervals, and the smallest tasks have about 160–180 questions, so their margins are roughly ±6–8 points and many model gaps fall inside that. The ablations are thin: the sampling-interval and window sweeps only tune settings, and nothing isolates EventGPT+'s dynamic bin selection. Train–test separation is only partial, since "most" samples come from different sources, and only the spatial subset has scene-level separation

Questions for authors

none

4 comments
AuthorsOpenReview ↗

Response to Reviewer ymcp (I/III)

We appreciate the reviewer’s positive note on EventBench’s community impact. Our core contribution is the full open benchmark infrastructure: dataset, baselines, evaluation code, and the Codabench leaderboard [1]. We are not introducing a new state-of-the-art model here. The baseline numbers we report serve as reproducible references for follow-up work. Its real-world uptake is shown by more than 10,000 downloads and over 100 registered teams across the two challenge tracks.

W1: Evaluation protocol and baselines

Sampling. Video MLLMs take 1 FPS frames rendered from event data, while event-based MLLMs directly consume raw event streams (lines 303–309). When sampling at 2 FPS, Qwen2.5-VL and InternVL-3.5 only pick up 0.2 and 0.3 percentage points, respectively. Even with the 0.5 s interval, EventGPT+(L) still outperforms the strongest video baseline by 8.4 points. This ranking stays consistent across all four sampling intervals.

Model0.5 s1.0 s1.5 s2.0 s
Qwen2.5-VL49.749.548.948.7
InternVL-3.556.556.255.955.6
EventGPT62.663.562.462.2
EventGPT+(L)64.965.564.363.9

Fine-tuning. We include closed-source models only as API baselines, since we do not have access to fine-tune them. For all open models, we ran fine-tuning on EQA-1.4M following each model’s native input format (lines 305–312). Table 3 shows Qwen2.5-VL and InternVL-3.5 improve across all eight tasks. For object counting specifically, their scores jump from 10.8% to 59.9% and 16.9% to 61.1%, respectively.

Paired RGB. We have paired RGB videos for the subset converted via v2e, but the native event datasets we use do not come with matching RGB footage. To keep training data and supervision consistent across all models, we use the same EQA-1.4M samples. Video MLLMs receive rendered event frames, while event-based models take raw event streams (lines 305–309).

AuthorsOpenReview ↗

Response to Reviewer ymcp (II/III)

W2: Annotation reliability and data separation

GPT-5's role. We used GPT-5 to draft the multiple-choice DU and CR questions, feeding it paired RGB-event data and existing annotations. We first filtered videos to keep only those with meaningful motion, then ran them through our quality classifier trained on 10,000 manually labeled samples (lines 156–159). Human annotators then verified each item: every question must be solvable from the event stream alone and have exactly one correct answer (lines 181–183). After excluding DU and CR, the models shown below retain the same ranking.

ModelAll 8 tasksExcluding DU and CR
GPT-545.137.0
Qwen2.5-VL49.545.1
InternVL-3.556.250.3
EventGPT63.558.7
EventGPT+(L)65.561.0

Train–test separation. We separate EQA-1.4M and EventBench at the sample level for all tasks (lines 159–160). The word “most” refers to our use of distinct source datasets. For spatial reasoning tasks, we go further and enforce separation at the scene level (line 161).

AuthorsOpenReview ↗

Response to Reviewer ymcp (III/III)

W3: Spatial reasoning, statistical precision and setting

SR reference. SR contains two-, three-, and four-candidate questions. The weighted uniform-random baseline is 35.91%, placing the reported 53.2% accuracy 17.3 percentage points above uniform guessing. The full dataset, annotations, and evaluation code are openly released, allowing independent verification of these counts and reference scores.

Candidate relations234All SR
Questions9997105301
Uniform-random expectation (%)50.0033.3325.0035.91

Test-set size. Suitable event recordings are scarce, so we prioritize stringent filtering, human verification, and task coverage. E-OCR spans seven scene categories, and CR contains 180 QA pairs from 101 source-samples identifiers. These task subsets are comparable in scale to those in BLINK [2], which includes task-specific test sets with 120–143 questions.

We ran EventGPT-Plus 2B on the complete CR set of 180 questions using five different decoding seeds. With temperature set to 0.3, accuracy fell between 74.44% and 76.67%, averaging 75.67% with a standard deviation of 0.91 points. All five greedy decoding runs returned exactly the same predictions. These numbers confirm that the overall CR accuracy stays stable across the decoding seeds we tested.

Decoding settingSeed 0Seed 1Seed 2Seed 3Seed 4Mean accuracySD (pp)
Greedy78.378.378.378.378.378.30.00
Sampling (temperature 0.3)75.076.174.476.776.175.70.91

Dynamic bin selection. Our baseline offers a lightweight, resource-efficient reference. We use dynamic bin selection by default so evaluation stays practical. Without this selection step, processing long event streams would demand excessive GPU memory. EventBench is fully open, and users can plug in their own models and customize binning strategies.

References

[1] Xu et al. Codabench: Flexible, easy-to-use, and reproducible meta-benchmark platform. Patterns, 2022.

[2] Fu et al. BLINK: Multimodal Large Language Models Can See but Not Perceive. ECCV, 2024.

AuthorsOpenReview ↗

A Kind Reminder

We would like to kindly follow up on our rebuttal, as the discussion period is approaching its end. We hope our additional experiments and clarifications have addressed your concerns. We would greatly appreciate any further feedback you may have before the discussion closes. Thank you again for your time and consideration.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Contributions, Rebuttal Evidence, and Discussion

Dear AC, SAC, and PC,

Thank you for your time throughout the review process. Below we summarize the main points from the reviews and discussion.

1. What the reviewers valued

  • ymcp saw EventBench’s role in bringing the event-camera MLLM community together as the main reason to accept the paper.
  • 9vFi recognized the need for a unified benchmark and the usefulness of its spatial reasoning tasks.
  • z8P4 highlighted the eight-task coverage, particularly the underexplored spatial reasoning tasks, and the value of EQA-1.4M and the cross-model evaluations.

EventBench provides datasets, baselines, evaluation code, and a Codabench platform. More than 100 teams have used it in a completed public competition.

2. Our responses and additional results

  • Evaluation robustness. Model rankings remained consistent across the existing sampling study and after excluding GPT-assisted DU/CR questions. We also reported separate real-event results, additional distance-error metrics, and decoding stability across five seeds.
  • Train–test separation applies to all tasks. All tasks use sample-level separation; spatial tasks also use scene-level separation. The word “most” refers to the use of different source datasets, not partial sample separation (original submission, §2.1, lines 159–161).
  • The 11.6% text-matching category includes both E-OCR and SR. Their 162 and 301 questions total 463/4,001 ≈ 11.6%; E-OCR alone accounts for approximately 4.05%. For SR, the candidates are given in the question, and the answer must exactly match the reference direction label (response to 9vFi, W1).

We revised the PDF to explain the answer formats and remove repeated text. The object-visibility checks and counting sensitivity analysis remain to be completed; we have committed to reporting these results in the camera-ready version.

3. Where the discussion stands

9vFi raised the rating by one point and thanked us in the final reply. Earlier, the reviewer noted that the statistics were clearer and thanked us for explaining the contribution and how the spatial ground truth was generated. We have not received further discussion comments from ymcp or z8P4 as of this summary.

We hope you will consider EventBench’s contribution and its use by the community, together with our additional evidence and the reviewer’s updated assessment, when making the final decision.

Best regards,
Authors of Paper 17799

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.