- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Existing video benchmarks can reward shortcuts in answer options, question text, or partial visual evidence, making it difficult to determine whether high scores reflect video understanding. This paper introduces an attack pyramid to systematically examine these shortcuts across 115 benchmarks. It then screens a large pool of existing questions and constructs Video-Index, an 840-item meta-benchmark covering four capability groups. Experiments reveal widespread shortcut dependence in existing benchmarks and show that Video-Index remains challenging for both fixed-input models and tool-using agents.
Critical strengths
1 Large-scale benchmark analysis. This paper applies a unified audit to 115 video benchmarks, providing a broad view of shortcut dependence that is difficult to obtain from isolated dataset studies.
2 Interpretable evaluation. The attack pyramid distinguishes different sources of shortcuts, including answer options, question text, and temporal information, helping explain what a benchmark score actually measures.
3 A useful evaluation resource. Video-Index combines attack-based filtering with source provenance and reproducible selection rules, allowing the audit findings to inform the construction of new evaluations.
Critical weaknesses
1 The criterion for declaring a benchmark broken needs stronger justification. The paper uses a fixed five-percentage-point margin relative to a reference model, despite differences in benchmark size and difficulty.
2 Model-specific filtering may bias the evaluation. Some models participate in screening or answer verification and are later evaluated on the resulting benchmark.
3 The agent comparison is not sufficiently controlled. Agents can seek, crop, and inspect additional visual evidence, while fixed-input models receive a predefined set of frames.
4 The paper would benefit from further polishing to improve clarity and readability.
Questions for authors
1 How stable are the benchmark-breaking conclusions under different reference models and benchmark-specific confidence intervals?
2 Could the authors evaluate models that played no role in benchmark construction and add a human audit of the retained answers?
3 Does the agent advantage remain when the two protocols are matched for the amount of visual evidence and total inference budget?
5 comments
W2 / Q2. Evaluating models that took part in construction
We agree that screening with a single model can introduce bias, and we measured it by replacing the screening model and by evaluating new models outside construction.
The original screen runs Qwen3-VL-8B in the options-only, blind and one-frame steps and Qwen3-VL-2B in the two 32-frame steps. We re-ran all five steps on the 840 Video-Index items with three replacement screens, namely InternVL3.5 at the same two sizes and two stronger models that run every step. A stronger model removes more items. Under every replacement screen, the six highest-ranked systems keep their order and the rank agreement is at least 0.95.
| Replacement screen | Options only | Blind | One frame | 32 frames | Shuffled 32 frames | Any step | Rank agreement |
|---|---|---|---|---|---|---|---|
| InternVL3.5-8B and InternVL3.5-2B | 1.0% | 0.7% | 5.0% | 3.1% | 2.6% | 9.6% | 0.99 |
| Qwen3-VL-32B | 1.1% | 1.5% | 4.6% | 19.9% | 13.6% | 23.5% | 0.95 |
| Gemma 3 27B | 2.6% | 4.2% | 5.1% | 8.9% | 8.8% | 17.3% | 0.96 |
The table above lists the share of the 840 Video-Index items that each step would remove. Rank agreement is Spearman ρ between the ranking of the 19 systems on all 840 items and on the items that the screen keeps.
Qwen3-VL, which screens the pool, is already excluded from the ranking-agreement statistics. We also evaluated four more models that played no part in construction.
| New system outside construction | Protocol | Video | Blind |
|---|---|---|---|
| Gemini 4 Argon | Agent | 80.4% | – |
| Gemini 4 Argon | Fixed input | 76.8% | – |
| InternVL3.5-30B-A3B | Fixed input | 21.4% | 4.9% |
| GLM-4.5V | Fixed input | 19.0% | 5.8% |
| Gemma 3 27B | Fixed input | 12.7% | 8.2% |
For the human audit, a person checked the items against the original videos. The audit removed 59 items whose ground truth is in doubt and corrected 4 answers. On the remaining 781 items, the 19 systems keep the order they have on all 840 items, with no pair reversed.
The accuracy of all 19 systems on the items that each screen keeps and on the 781 items that remain after the audit is as follows, with the rank in parentheses.
| System | Protocol | All 840 items | Kept by InternVL3.5 (759) | Kept by Qwen3-VL-32B (643) | Kept by Gemma 3 27B (695) | After the audit (781) |
|---|---|---|---|---|---|---|
| Gemini 4 Argon | Agent | 80.4% (1) | 79.6% (1) | 75.7% (1) | 78.1% (1) | 86.8% (1) |
| GPT-6-Astra | Agent | 79.3% (2) | 78.2% (2) | 74.2% (2) | 77.5% (2) | 86.6% (2) |
| Gemini 4 Argon | Fixed input | 76.8% (3) | 75.9% (3) | 71.1% (3) | 74.0% (3) | 82.2% (3) |
| Claude Fable 5.1 | Agent | 71.5% (4) | 70.5% (4) | 65.9% (4) | 69.3% (4) | 78.1% (4) |
| Claude Opus 5 | Agent | 70.6% (5) | 69.8% (5) | 63.3% (5) | 67.0% (5) | 76.3% (5) |
| Claude Opus 5 | Fixed input | 56.8% (6) | 55.1% (6) | 49.6% (6) | 51.1% (6) | 60.3% (6) |
| Qwen3-VL-32B | Fixed input | 23.0% (7) | 20.6% (7) | 8.2% (10) | 17.1% (8) | 25.0% (7) |
| InternVL3.5-30B-A3B | Fixed input | 21.4% (8) | 18.6% (8) | 10.9% (8) | 17.3% (7) | 22.7% (8) |
| Molmo2-8B | Fixed input | 19.0% (9) | 16.7% (9) | 11.0% (7) | 16.1% (9) | 20.5% (9) |
| GLM-4.5V | Fixed input | 19.0% (9) | 16.2% (10) | 10.6% (9) | 14.7% (10) | 20.0% (10) |
| InternVL3.5-8B | Fixed input | 15.2% (11) | 11.7% (13) | 6.8% (15) | 11.4% (11) | 16.4% (11) |
| LLaVA-OneVision-2-8B | Fixed input | 14.2% (12) | 12.1% (11) | 7.3% (12) | 9.9% (12) | 15.1% (12) |
| Gemma 3 27B | Fixed input | 12.7% (13) | 12.1% (11) | 8.1% (11) | 5.6% (19) | 13.6% (13) |
| InternVideo2.5-8B | Fixed input | 12.4% (14) | 10.0% (16) | 4.8% (18) | 9.2% (13) | 13.4% (14) |
| VideoLLaMA3-7B | Fixed input | 12.4% (14) | 9.9% (17) | 7.3% (12) | 8.8% (15) | 13.1% (16) |
| Video-XL-2 | Fixed input | 12.4% (14) | 10.3% (14) | 7.0% (14) | 9.1% (14) | 13.3% (15) |
| Cambrian-S-7B | Fixed input | 11.3% (17) | 10.1% (15) | 5.1% (17) | 7.8% (16) | 12.2% (17) |
| Qwen3-VL-8B | Fixed input | 9.5% (18) | 7.8% (18) | 3.7% (19) | 7.3% (17) | 10.2% (18) |
| VideoChat3-4B | Fixed input | 9.4% (19) | 7.5% (19) | 5.3% (16) | 6.5% (18) | 10.2% (18) |
A screen removes every item that its own model answers correctly, so Qwen3-VL-32B and Gemma 3 27B fall in the column of their own screen. Systems with the same accuracy share a rank.
The results on the screening models and on the new models are in the revision (Tables 56, 57, 59 and 60 in Appendix W). In future work, we will design a reliable automated pipeline for answer verification.
Together, these results indicate that the ranking of systems on Video-Index is robust to the models used in construction. The ranking holds under other screening models, for models outside construction, and after the answer audit, and the screening model itself, which is lowered by construction, is excluded from the ranking-agreement statistics.
W3 / Q3. Agent advantage under matched evidence and budget
Thank you for the suggestion. Table 1 compares two evaluation protocols, not two kinds of model. Each agent runs in its vendor's own harness (Codex for GPT-6-Astra, Claude Code for Claude, Gemini CLI for Gemini) with the video file and a shell.
To compare the two protocols on the same visual evidence, we ran three matched settings with Claude Opus 5 on 210 items. These are Claude Opus 5 in Claude Code restricted to the frames of the fixed protocol, a single call from the same frames at the same maximum reasoning effort, and a single call from exactly the images that it read as an agent in the original run. The results are shown below and in Figure V2 and Table 50 (Appendix V).
| Step | Accuracy | Gain over the row above |
|---|---|---|
| Fixed input, as published | 53.3 | |
| Same frames, current API call setup | 55.7 | +2.4 |
| Maximum reasoning effort | 60.5 | +4.8 |
| The images the agent chose instead of uniform frames | 65.2 | +4.8 |
| Inspecting over several turns (agent restricted to the same frames) | 69.5 | +4.3 |
With identical visual evidence and effort, Claude Opus 5 in Claude Code is 9.0 points higher than the single call. The agent's advantage therefore does not come from more pixels or frames, and longer reasoning explains about a third of it. The rest comes in roughly equal parts from choosing which frames and regions to inspect and from inspecting them over several turns. Restricting the visual input changes agent accuracy by only 1.0 point.
We also gave the images that Claude Opus 5 chose as an agent to open-source models, in place of the uniform frames of the fixed protocol. The images that the agent chose improve only some of the open-source models, and no open-source model comes close to the closed-source model (Table 51, Appendix V). Gains in bold are significant, which holds for six of the twelve open-source models.
| Model | Uniform frames | Images chosen by the agent | Gain |
|---|---|---|---|
| Claude Opus 5 (closed-source, for reference) | 60.5% | 65.2% | +4.8 |
| Qwen3-VL-32B | 21.0% | 32.4% | +11.4 |
| GLM-4.5V | 17.1% | 24.3% | +7.1 |
| Gemma 3 27B | 11.0% | 19.5% | +8.6 |
| InternVL3.5-8B | 14.8% | 17.6% | +2.9 |
| Molmo2-8B | 18.6% | 17.1% | −1.4 |
| LLaVA-OneVision-2-8B | 12.4% | 17.1% | +4.8 |
| Qwen3-VL-8B | 6.7% | 14.8% | +8.1 |
| InternVideo2.5-8B | 8.6% | 14.3% | +5.7 |
| VideoLLaMA3-7B | 11.4% | 12.9% | +1.4 |
| Video-XL-2 | 10.5% | 10.0% | −0.5 |
| Cambrian-S-7B | 8.6% | 9.0% | +0.5 |
| VideoChat3-4B | 7.6% | 8.6% | +1.0 |
With its visual input restricted to the fixed frames, Claude Opus 5 in Claude Code (69.5%) stays far above every open-source model on the same 210 items (6.7–21.0% for the 13 open models under the fixed protocol). Gemini 4 Argon shows the same direction on all 840 items, where it scores 80.4% as an agent in Gemini CLI and 76.8% from the fixed frames (+3.6 points).
Both protocols receive the same visual information, and the agents use less of it. The fixed protocol sends 187 frames per item on average, and the agents read 7 to 19 images per answer. We report the mean cost per answer of every system for reference (Table 52, Appendix V).
| System | Protocol | Mean cost per answer in USD |
|---|---|---|
| Claude Opus 5 | Fixed input | 0.12 |
| Gemini 4 Argon | Fixed input | 0.42 |
| Gemini 3.5 Flash-Lite | Agent | 0.08 |
| Gemini 3.1 Pro | Agent | 0.56 |
| Claude Opus 5 | Agent | 1.40 |
| Claude Fable 5.1 | Agent | 1.80 |
| GPT-6-Astra | Agent | 0.29 |
| Gemini 4 Argon | Agent | 0.10 |
W4. Clarity and readability
Thank you for the suggestion. We have submitted a revised PDF in which all changes are marked in blue, and we will revise the writing further.
W1 / Q1. Justification and stability of the breaking criterion
Thank you for this question. Benchmarks differ in size, so we audit every benchmark on a fixed random sample of 300 items (or all items when it has fewer, and only 4 benchmarks have fewer than 295 items). For the threshold, Appendix P recounts all levels at δ = 3% and 8%, and with Claude Opus 5 as the reference model. We acknowledge that the threshold changes how many benchmarks sit at each level. The relative order of the benchmarks is largely preserved, and our findings hold in every setting (Figure P1a).
In addition, for benchmark-specific confidence, we ran an item-level bootstrap (1,000 resamples within each benchmark, recomputing s*, c and every item-level attacker). Whether a benchmark breaks before or after video is unchanged in at least 90% of resamples for 86 of the 115 benchmarks (Figure P1b), and the difference between newer and older benchmarks stays positive in 97% of resamples (Figure P1c). The revision adds two stability scores to the report card of every benchmark (Appendix Z), namely the share of resamples that reproduce its breaking level and the share that fall on the same side of visual input.
Together, these results show that the breaking level reflects the benchmark itself and not the choice of threshold, reference model or item sample, so the attack pyramid gives a reliable diagnosis of where a benchmark can be bypassed.