- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
VideoGAIA introduces 271 tasks across six domains to evaluate assistants that combine video evidence with external information retrieval. Tasks are generated from visual clues, filtered using models, and independently verified by human annotators. Twenty multimodal models are evaluated using a common agent loop with web search, webpage reading, and video reinspection. The highest reported accuracy is 58.30%. The paper also analyzes difficulty, tool use, and failure trajectories.
Critical strengths
-
The tasks connect visual recognition with information seeking in a concrete way (§§3.1–3.2). Models need to identify relevant clues in the video and use them to retrieve an external fact. The modality and tool-necessity filters help support this task design.
-
The human verification is a strong part of the construction process. Three annotators independently solve each candidate task, and tasks approved by two annotators receive further review (§3.2). The evaluation of 20 models with common prompts, tools, and interaction limits also provides a useful reference (§4.1).
-
Figure 8 is a helpful example of what makes the tasks difficult. Recognizing Honda branding is insufficient to identify the engine displacement; the successful trajectory considers candidate bike models and checks a more specific visual cue. The example makes the interaction between visual recognition and external verification easy to understand.
Critical weaknesses
-
I would like a clearer picture of the range of tasks produced by the construction process (§§3.1–3.3). The pipeline centers on identifying visual clues and retrieving related facts. This is a useful task family, but the six topical categories do not tell us how varied the actual problem-solving requirements are. A clearer account of the task mix would help readers understand what VideoGAIA primarily measures.
-
Since every model already receives 20 uniformly sampled frames, it remains unclear how much performance depends on revisiting the video rather than identifying clues from the initial frames and searching the web (§4.1). Figure 8 illustrates the value of reinspection in one case, but all leaderboard results use the full agent loop. A comparison using only the initial frames and web tools would help clarify how important video reinspection is across the benchmark.
-
A smaller reporting concern is that scores are averaged over three runs, but the variation is not reported, making small gaps in Table 1 difficult to interpret. The grading rules are provided in Appendix B.2; it would also help to know whether the judge’s decisions on model answers were checked against human judgments.
Questions for authors
-
Could you describe the main evidence or reasoning requirements represented in the final task set, beyond the six topical categories?
-
For one or two representative models, could you compare the full agent loop with a condition that retains the initial 20 frames and web tools but disables video reinspection?
-
Could you report the variation across the three runs for the leading models and describe any manual checks of the automatic grader’s decisions?
5 comments
Response to Reviewer gCKX [Part 2]
W3 & Q3. Run-to-Run Variance and Grader Validation
W3. A smaller reporting concern is that scores are averaged over three runs, but the variation is not reported, making small gaps in Table 1 difficult to interpret. The grading rules are provided in Appendix B.2; it would also help to know whether the judge’s decisions on model answers were checked against human judgments.
Q3. Could you report the variation across the three runs for the leading models and describe any manual checks of the automatic grader’s decisions?
Thank you for your comment and question. We report the overall accuracy (%) of three independent runs as follows.
| Model | Run 1 (Table 1) | Run 2 | Run 3 | Mean +/- Std |
|---|---|---|---|---|
| Seed2.0-Pro | 58.30 | 57.93 | 56.46 | 57.56 +/- 0.98 |
| Qwen3.7-Plus | 56.09 | 57.56 | 56.09 | 56.58 +/- 0.85 |
| Kimi-K3 | 54.61 | 55.35 | 55.35 | 55.10 +/- 0.43 |
| GPT-5.5 | 53.14 | 56.09 | 56.09 | 55.10 +/- 1.70 |
| Kimi-K2.6 | 52.77 | 53.51 | 52.40 | 52.89 +/- 0.56 |
| GPT-5.4 | 52.03 | 52.40 | 53.14 | 52.52 +/- 0.56 |
| Qwen3.5-Plus | 52.03 | 49.45 | 52.40 | 51.29 +/- 1.61 |
| GPT-5.2 | 50.92 | 50.18 | 50.92 | 50.68 +/- 0.43 |
| GLM-5V-Turbo | 50.18 | 48.71 | 52.40 | 50.43 +/- 1.86 |
| Qwen3.8-Max | 50.18 | 49.82 | 48.71 | 49.57 +/- 0.77 |
| Kimi-K2.5 | 49.82 | 49.45 | 49.45 | 49.57 +/- 0.21 |
| Gemini-3.1-Pro | 50.92 | 49.08 | 47.97 | 49.32 +/- 1.49 |
| MiMo-V2.5 | 48.71 | 48.34 | 50.55 | 49.20 +/- 1.19 |
| Qwen3.5-397B | 50.18 | 48.71 | 47.60 | 48.83 +/- 1.30 |
| Qwen3.6-Plus | 49.08 | 47.23 | 48.71 | 48.34 +/- 0.98 |
| Claude-Opus-4.6 | 47.97 | 47.60 | 48.71 | 48.09 +/- 0.56 |
| GLM-4.6V | 46.86 | 44.28 | 47.97 | 46.37 +/- 1.89 |
| Gemini-3.5-Flash | 43.91 | 41.70 | 44.65 | 43.42 +/- 1.54 |
| Claude-Opus-4.8 | 36.90 | 38.01 | 38.01 | 37.64 +/- 0.64 |
| Claude-Opus-4.7 | 34.69 | 29.89 | 33.95 | 32.84 +/- 2.58 |
The standard deviation ranges from 0.21 to 2.58 points, and the Spearman correlation between the model rankings of any two runs is at least 0.88. Seed2.0-Pro ranks first in every run, and no model reaches 60% in any run. We will report the mean +/- std for all columns of Table 1 in the revision.
Regarding the grader, we did not conduct a separate manual check, since the judge receives the question, the prediction, and a short human-verified reference answer (3.73 tokens on average). We instead re-graded the Run 1 predictions of three leading models with two judges from other model families, as follows.
| Model | GPT-5.5 Judge | DeepSeek-V4-Pro Judge | Claude-Opus-4.8 Judge |
|---|---|---|---|
| Seed2.0-Pro | 158/271 | 156/271 | 157/271 |
| Qwen3.7-Plus | 152/271 | 152/271 | 151/271 |
| Gemini-3.1-Pro | 138/271 | 138/271 | 138/271 |
The three judges differ by at most two of the 271 tasks and yield the same ranking, which supports the reliability of the LLM judge. This protocol also follows SimpleQA (Wei et al., 2024), Humanity's Last Exam (Phan et al., 2025), and BrowseComp (Wei et al., 2025), and Zheng et al. (2023) and Thakur et al. (2025) show that strong LLM judges agree well with human judgments. We will add a manual validation of the judge in future work.
References
- Long Phan, Alice Gatti, Ziwen Han, et al. Humanity's Last Exam. arXiv preprint arXiv:2501.14249, 2025.
- Aman Singh Thakur, Kartik Choudhary, Venkat Srinik Ramayapally, et al. Judging the Judges: Evaluating Alignment and Vulnerabilities in LLMs-as-Judges. In Proceedings of the Fourth Workshop on Generation, Evaluation and Metrics (GEM²), 2025.
- Jason Wei, Nguyen Karina, Hyung Won Chung, et al. Measuring Short-Form Factuality in Large Language Models. arXiv preprint arXiv:2411.04368, 2024.
- Jason Wei, Zhiqing Sun, Spencer Papay, et al. BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agents. arXiv preprint arXiv:2504.12516, 2025.
- Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, et al. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. In Advances in Neural Information Processing Systems (NeurIPS), Datasets and Benchmarks Track, 2023.
Thank you again for your constructive feedback, which helped us improve the quality of this work. If our responses have addressed your concerns, we would be grateful if you could consider improving your rating. If you have any other questions or concerns that remain unresolved, we would be very happy to address them.
Follow-up on Our Response
Dear Reviewer gCKX,
Thank you again for your constructive feedback on our submission. In our response, we described the evidence and reasoning requirements of all 271 tasks, showed that disabling video reinspection lowers accuracy by 7.01 points for Qwen3.8-Max and 5.90 points for Gemini-3.1-Pro, and reported three-run results together with cross-judge validation.
As the discussion period ends on November 18 (AoE), we would be grateful if you could let us know whether our responses have addressed your concerns. If they have, we would appreciate it if you could consider improving your rating. If any questions remain, we would be very happy to address them before the deadline.
Best regards,
The Authors of Submission 1955, VideoGAIA
Response to Reviewer gCKX [Part 1]
We sincerely thank you for your time and constructive suggestions. In the following response, we try to address your concerns point by point.
W1 & Q1. Task Mix beyond the Six Topical Categories
Thank you for your comment and question. We describe the evidence and reasoning requirements of all 271 tasks as follows.
The topical categories reflect only part of this variation, since text, logo, and object evidence appears in all six categories. In addition, 72 questions (26.6%) refer to a specific moment or the order of events in the clip. We will add this breakdown to §3.3.
W2 & Q2. Contribution of Video Reinspection
Thank you for your comment and question. We compare the full agent loop with a variant that keeps the initial 20 frames and web tools but disables video reinspection, as follows.
Disabling video reinspection lowers overall accuracy by 7.01 points for Qwen3.8-Max and 5.90 points for Gemini-3.1-Pro. The drop appears in five of the six categories for Qwen3.8-Max and in all six for Gemini-3.1-Pro. We will add this ablation to the revised paper.