acceptodds
Under review as a conference paper at ICLR 2027

Video-Index: A Curated Meta-Benchmark for Video Understanding Evaluation

10 views

Abstract

A video benchmark should reward the capability it claims to measure, yet models can exploit answer options, question text, or partial visual evidence. We introduce the attack pyramid, five levels of shortcut attacks with increasing access to each item, and audit 115 video benchmarks with it. On 35 benchmarks, attackers that never see a frame approach full-video accuracy. On 51 benchmarks with temporal probes, shuffled frames keep a median 96% of full-video accuracy. Near-duplicate questions make up at least half the items in 63 benchmarks. We screen 505,518 question–answer pairs from 112 of them into an audited pool. Agents turn evaluation requests into specifications, and a deterministic selector with a red-team gate composes reproducible benchmarks. We release Video-Index, the 210 hardest verified items under these attacks in each of four capability groups, 840 items from 76 sources. With the same fixed input, Claude Opus 5 outscores every open-source model by over 37 percentage points, and agent tools add about 20 more, yet all systems leave room to improve efficiency and accuracy.

Primary area:
datasets and benchmarks
Keywords:
video understanding, video benchmarks, benchmark auditing, shortcut learning, multimodal large language models, evaluation

Reviews

Among:
Shape:
1234this paper, Top 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.67 over 3 reviews, above 99% of the others.
Reviewer_HsMQOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Existing video benchmarks can reward shortcuts in answer options, question text, or partial visual evidence, making it difficult to determine whether high scores reflect video understanding. This paper introduces an attack pyramid to systematically examine these shortcuts across 115 benchmarks. It then screens a large pool of existing questions and constructs Video-Index, an 840-item meta-benchmark covering four capability groups. Experiments reveal widespread shortcut dependence in existing benchmarks and show that Video-Index remains challenging for both fixed-input models and tool-using agents.

Critical strengths

1 Large-scale benchmark analysis. This paper applies a unified audit to 115 video benchmarks, providing a broad view of shortcut dependence that is difficult to obtain from isolated dataset studies.

2 Interpretable evaluation. The attack pyramid distinguishes different sources of shortcuts, including answer options, question text, and temporal information, helping explain what a benchmark score actually measures.

3 A useful evaluation resource. Video-Index combines attack-based filtering with source provenance and reproducible selection rules, allowing the audit findings to inform the construction of new evaluations.

Critical weaknesses

1 The criterion for declaring a benchmark broken needs stronger justification. The paper uses a fixed five-percentage-point margin relative to a reference model, despite differences in benchmark size and difficulty.

2 Model-specific filtering may bias the evaluation. Some models participate in screening or answer verification and are later evaluated on the resulting benchmark.

3 The agent comparison is not sufficiently controlled. Agents can seek, crop, and inspect additional visual evidence, while fixed-input models receive a predefined set of frames.

4 The paper would benefit from further polishing to improve clarity and readability.

Questions for authors

1 How stable are the benchmark-breaking conclusions under different reference models and benchmark-specific confidence intervals?

2 Could the authors evaluate models that played no role in benchmark construction and add a human audit of the retained answers?

3 Does the agent advantage remain when the two protocols are matched for the amount of visual evidence and total inference budget?

5 comments

W1 / Q1. Justification and stability of the breaking criterion

Thank you for this question. Benchmarks differ in size, so we audit every benchmark on a fixed random sample of 300 items (or all items when it has fewer, and only 4 benchmarks have fewer than 295 items). For the threshold, Appendix P recounts all levels at δ = 3% and 8%, and with Claude Opus 5 as the reference model. We acknowledge that the threshold changes how many benchmarks sit at each level. The relative order of the benchmarks is largely preserved, and our findings hold in every setting (Figure P1a).

SettingBreak before videoBenchmark pairs in the same order as the main audit
δ = 5% (main audit)35
δ = 3%3390%
δ = 8%4980%
Claude Opus 5 as the reference, δ = 5%3179%

In addition, for benchmark-specific confidence, we ran an item-level bootstrap (1,000 resamples within each benchmark, recomputing s*, c and every item-level attacker). Whether a benchmark breaks before or after video is unchanged in at least 90% of resamples for 86 of the 115 benchmarks (Figure P1b), and the difference between newer and older benchmarks stays positive in 97% of resamples (Figure P1c). The revision adds two stability scores to the report card of every benchmark (Appendix Z), namely the share of resamples that reproduce its breaking level and the share that fall on the same side of visual input.

Together, these results show that the breaking level reflects the benchmark itself and not the choice of threshold, reference model or item sample, so the attack pyramid gives a reliable diagnosis of where a benchmark can be bypassed.

W2 / Q2. Evaluating models that took part in construction

We agree that screening with a single model can introduce bias, and we measured it by replacing the screening model and by evaluating new models outside construction.

The original screen runs Qwen3-VL-8B in the options-only, blind and one-frame steps and Qwen3-VL-2B in the two 32-frame steps. We re-ran all five steps on the 840 Video-Index items with three replacement screens, namely InternVL3.5 at the same two sizes and two stronger models that run every step. A stronger model removes more items. Under every replacement screen, the six highest-ranked systems keep their order and the rank agreement is at least 0.95.

Replacement screenOptions onlyBlindOne frame32 framesShuffled 32 framesAny stepRank agreement
InternVL3.5-8B and InternVL3.5-2B1.0%0.7%5.0%3.1%2.6%9.6%0.99
Qwen3-VL-32B1.1%1.5%4.6%19.9%13.6%23.5%0.95
Gemma 3 27B2.6%4.2%5.1%8.9%8.8%17.3%0.96

The table above lists the share of the 840 Video-Index items that each step would remove. Rank agreement is Spearman ρ between the ranking of the 19 systems on all 840 items and on the items that the screen keeps.

Qwen3-VL, which screens the pool, is already excluded from the ranking-agreement statistics. We also evaluated four more models that played no part in construction.

New system outside constructionProtocolVideoBlind
Gemini 4 ArgonAgent80.4%–
Gemini 4 ArgonFixed input76.8%–
InternVL3.5-30B-A3BFixed input21.4%4.9%
GLM-4.5VFixed input19.0%5.8%
Gemma 3 27BFixed input12.7%8.2%

For the human audit, a person checked the items against the original videos. The audit removed 59 items whose ground truth is in doubt and corrected 4 answers. On the remaining 781 items, the 19 systems keep the order they have on all 840 items, with no pair reversed.

The accuracy of all 19 systems on the items that each screen keeps and on the 781 items that remain after the audit is as follows, with the rank in parentheses.

SystemProtocolAll 840 itemsKept by InternVL3.5 (759)Kept by Qwen3-VL-32B (643)Kept by Gemma 3 27B (695)After the audit (781)
Gemini 4 ArgonAgent80.4% (1)79.6% (1)75.7% (1)78.1% (1)86.8% (1)
GPT-6-AstraAgent79.3% (2)78.2% (2)74.2% (2)77.5% (2)86.6% (2)
Gemini 4 ArgonFixed input76.8% (3)75.9% (3)71.1% (3)74.0% (3)82.2% (3)
Claude Fable 5.1Agent71.5% (4)70.5% (4)65.9% (4)69.3% (4)78.1% (4)
Claude Opus 5Agent70.6% (5)69.8% (5)63.3% (5)67.0% (5)76.3% (5)
Claude Opus 5Fixed input56.8% (6)55.1% (6)49.6% (6)51.1% (6)60.3% (6)
Qwen3-VL-32BFixed input23.0% (7)20.6% (7)8.2% (10)17.1% (8)25.0% (7)
InternVL3.5-30B-A3BFixed input21.4% (8)18.6% (8)10.9% (8)17.3% (7)22.7% (8)
Molmo2-8BFixed input19.0% (9)16.7% (9)11.0% (7)16.1% (9)20.5% (9)
GLM-4.5VFixed input19.0% (9)16.2% (10)10.6% (9)14.7% (10)20.0% (10)
InternVL3.5-8BFixed input15.2% (11)11.7% (13)6.8% (15)11.4% (11)16.4% (11)
LLaVA-OneVision-2-8BFixed input14.2% (12)12.1% (11)7.3% (12)9.9% (12)15.1% (12)
Gemma 3 27BFixed input12.7% (13)12.1% (11)8.1% (11)5.6% (19)13.6% (13)
InternVideo2.5-8BFixed input12.4% (14)10.0% (16)4.8% (18)9.2% (13)13.4% (14)
VideoLLaMA3-7BFixed input12.4% (14)9.9% (17)7.3% (12)8.8% (15)13.1% (16)
Video-XL-2Fixed input12.4% (14)10.3% (14)7.0% (14)9.1% (14)13.3% (15)
Cambrian-S-7BFixed input11.3% (17)10.1% (15)5.1% (17)7.8% (16)12.2% (17)
Qwen3-VL-8BFixed input9.5% (18)7.8% (18)3.7% (19)7.3% (17)10.2% (18)
VideoChat3-4BFixed input9.4% (19)7.5% (19)5.3% (16)6.5% (18)10.2% (18)

A screen removes every item that its own model answers correctly, so Qwen3-VL-32B and Gemma 3 27B fall in the column of their own screen. Systems with the same accuracy share a rank.

The results on the screening models and on the new models are in the revision (Tables 56, 57, 59 and 60 in Appendix W). In future work, we will design a reliable automated pipeline for answer verification.

Together, these results indicate that the ranking of systems on Video-Index is robust to the models used in construction. The ranking holds under other screening models, for models outside construction, and after the answer audit, and the screening model itself, which is lowered by construction, is excluded from the ranking-agreement statistics.

W3 / Q3. Agent advantage under matched evidence and budget

Thank you for the suggestion. Table 1 compares two evaluation protocols, not two kinds of model. Each agent runs in its vendor's own harness (Codex for GPT-6-Astra, Claude Code for Claude, Gemini CLI for Gemini) with the video file and a shell.

To compare the two protocols on the same visual evidence, we ran three matched settings with Claude Opus 5 on 210 items. These are Claude Opus 5 in Claude Code restricted to the frames of the fixed protocol, a single call from the same frames at the same maximum reasoning effort, and a single call from exactly the images that it read as an agent in the original run. The results are shown below and in Figure V2 and Table 50 (Appendix V).

StepAccuracyGain over the row above
Fixed input, as published53.3
Same frames, current API call setup55.7+2.4
Maximum reasoning effort60.5+4.8
The images the agent chose instead of uniform frames65.2+4.8
Inspecting over several turns (agent restricted to the same frames)69.5+4.3

With identical visual evidence and effort, Claude Opus 5 in Claude Code is 9.0 points higher than the single call. The agent's advantage therefore does not come from more pixels or frames, and longer reasoning explains about a third of it. The rest comes in roughly equal parts from choosing which frames and regions to inspect and from inspecting them over several turns. Restricting the visual input changes agent accuracy by only 1.0 point.

We also gave the images that Claude Opus 5 chose as an agent to open-source models, in place of the uniform frames of the fixed protocol. The images that the agent chose improve only some of the open-source models, and no open-source model comes close to the closed-source model (Table 51, Appendix V). Gains in bold are significant, which holds for six of the twelve open-source models.

ModelUniform framesImages chosen by the agentGain
Claude Opus 5 (closed-source, for reference)60.5%65.2%+4.8
Qwen3-VL-32B21.0%32.4%+11.4
GLM-4.5V17.1%24.3%+7.1
Gemma 3 27B11.0%19.5%+8.6
InternVL3.5-8B14.8%17.6%+2.9
Molmo2-8B18.6%17.1%−1.4
LLaVA-OneVision-2-8B12.4%17.1%+4.8
Qwen3-VL-8B6.7%14.8%+8.1
InternVideo2.5-8B8.6%14.3%+5.7
VideoLLaMA3-7B11.4%12.9%+1.4
Video-XL-210.5%10.0%−0.5
Cambrian-S-7B8.6%9.0%+0.5
VideoChat3-4B7.6%8.6%+1.0

With its visual input restricted to the fixed frames, Claude Opus 5 in Claude Code (69.5%) stays far above every open-source model on the same 210 items (6.7–21.0% for the 13 open models under the fixed protocol). Gemini 4 Argon shows the same direction on all 840 items, where it scores 80.4% as an agent in Gemini CLI and 76.8% from the fixed frames (+3.6 points).

Both protocols receive the same visual information, and the agents use less of it. The fixed protocol sends 187 frames per item on average, and the agents read 7 to 19 images per answer. We report the mean cost per answer of every system for reference (Table 52, Appendix V).

SystemProtocolMean cost per answer in USD
Claude Opus 5Fixed input0.12
Gemini 4 ArgonFixed input0.42
Gemini 3.5 Flash-LiteAgent0.08
Gemini 3.1 ProAgent0.56
Claude Opus 5Agent1.40
Claude Fable 5.1Agent1.80
GPT-6-AstraAgent0.29
Gemini 4 ArgonAgent0.10

W4. Clarity and readability

Thank you for the suggestion. We have submitted a revised PDF in which all changes are marked in blue, and we will revise the writing further.

Reviewer_oC8vOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper examines whether existing video understanding benchmarks actually measure the capabilities they claim to evaluate. The authors introduce an attack pyramid that tests different shortcuts, such as answering questions without videos or using shuffled frames. By analyzing 115 video benchmarks, they find that many can be solved without fully understanding the video content. Based on these findings, the authors introduce Video-Index, a new benchmark containing challenging questions selected from existing datasets.

Critical strengths
  • I think one of the main strengths of this paper is that it systematically examines what existing video benchmarks actually measure before building a new benchmark. I find this approach novel and meaningful, and I think it is an important contribution to the field. These findings raise important questions about how reliable current video understanding evaluations are.

  • Another strength is the construction of Video-Index. The authors carefully filter existing questions using different shortcut attacks and provide a more challenging benchmark. I think this is a useful contribution for evaluating future VideoLLMs.

Critical weaknesses
  • It is somewhat unclear whether Video-Index aims to measure video understanding more reliably or simply collect questions that current models find difficult. For example, the authors remove questions that Qwen3-VL-2B answers correctly using the original video, but this does not necessarily mean those questions rely on shortcuts. I'm also concerned that filtering questions based on a specific model could introduce selection bias into the benchmark.

  • Another concern is the reliability of answer verification. Claude Opus 5 is used for several parts of the benchmark construction, while also being evaluated on Video-Index. In addition, the paper only explicitly excludes unverifiable questions from shorter videos. I think independent human verification, or at least a clearer analysis of these unverifiable cases, would help establish the reliability of the benchmark.

Questions for authors
  • Could the authors clarify how many people participated, how many questions they answered, and how they watched the videos for human accuracy?

  • How sensitive is Video-Index to the models used for shortcut filtering? For example, would using different or stronger models lead to a different set of questions?

5 comments

W1. Goal of Video-Index and selection bias from the filter

Thank you for pointing out this ambiguity. The two aims are not alternatives, and Video-Index pursues both. The shortcut filters make its score reflect video understanding, and the difficulty filter keeps the test set challenging for current models. Sections 3 and 4 use the shortcut attacks to audit existing benchmarks. Section 5.1 then builds Video-Index, and its last step removes the items that Qwen3-VL-2B answers correctly from 32 frames, in the original or a shuffled order. This step is a difficulty criterion, and we do not claim that the removed items contain shortcuts. A recent small model stands for what is no longer hard, and removing what it already solves leaves a more challenging test set.

We agree that filtering with a specific model introduces an unavoidable bias, and we measured its size by replacing the model. With InternVL3.5-2B in place of Qwen3-VL-2B, the two 32-frame steps would remove only 3.1% and 2.6% of Video-Index, and all five steps together 9.6%.

We then scored the 19 systems with per-item answers on the 759 items that pass the InternVL3.5 re-screen and compared the ranking with the one on all 840 items (Table 60, Appendix W).

Rank on 840 itemsSystemProtocolAccuracy on 840 itemsAccuracy on the 759 itemsRank on the 759 items
1Gemini 4 ArgonAgent80.4%79.6%1
2GPT-6-AstraAgent79.3%78.2%2
3Gemini 4 ArgonFixed input76.8%75.9%3
4Claude Fable 5.1Agent71.5%70.5%4
5Claude Opus 5Agent70.6%69.8%5
6Claude Opus 5Fixed input56.8%55.1%6
7Qwen3-VL-32BFixed input23.0%20.6%7
8InternVL3.5-30B-A3BFixed input21.4%18.6%8
9Molmo2-8BFixed input19.0%16.7%9
9GLM-4.5VFixed input19.0%16.2%10
11InternVL3.5-8BFixed input15.2%11.7%13
12LLaVA-OneVision-2-8BFixed input14.2%12.1%11
13Gemma 3 27BFixed input12.7%12.1%11
14InternVideo2.5-8BFixed input12.4%10.0%16
14VideoLLaMA3-7BFixed input12.4%9.9%17
14Video-XL-2Fixed input12.4%10.3%14
17Cambrian-S-7BFixed input11.3%10.1%15
18Qwen3-VL-8BFixed input9.5%7.8%18
19VideoChat3-4BFixed input9.4%7.5%19

Systems with the same accuracy share a rank.

The ten highest-ranked systems keep their order (Spearman ρ = 0.99 over all 19), and the changes are between open models with fixed input in the lower half. Two stronger screens remove more items, and the six highest-ranked systems still keep their order. This shows that the screening model changes, to some extent, which items are finally kept, but it does not change the overall ordering of the evaluated models.

W2. Reliability of the answer verification

Thank you for this question. For consistency of construction and for cost, we used a single model, Claude Opus 5, for labeling and answer verification. We agree that verification by several models is more reliable and that a step relying on one model is sensitive to that model. We therefore added checks that do not depend on Claude. A human audit verified the answers again and removed the items in doubt, a human reviewer and agents completed the verification of the long videos that Claude Opus 5 could not confirm, and we tested whether verification by Claude Opus 5 affects its own score.

We verified the items again, and this verification was done by a person. Agents only flagged the items on which none of them selected the marked answer, and an author watched the original video of every flagged item. Every decision to keep, correct or remove a flagged item was made by a person against the original video. We removed the 59 items whose answer is in doubt and corrected 4 answers. The 19 systems keep their order on the remaining 781 items, with no pair reversed.

SystemProtocolAccuracy on 840 itemsRank on 840 itemsAccuracy on the 781 itemsRank on the 781 items
Gemini 4 ArgonAgent80.4%186.8%1
GPT-6-AstraAgent79.3%286.6%2
Gemini 4 ArgonFixed input76.8%382.2%3
Claude Fable 5.1Agent71.5%478.1%4
Claude Opus 5Agent70.6%576.3%5
Claude Opus 5Fixed input56.8%660.3%6
Qwen3-VL-32BFixed input23.0%725.0%7
InternVL3.5-30B-A3BFixed input21.4%822.7%8
Molmo2-8BFixed input19.0%920.5%9
GLM-4.5VFixed input19.0%920.0%10
InternVL3.5-8BFixed input15.2%1116.4%11
LLaVA-OneVision-2-8BFixed input14.2%1215.1%12
Gemma 3 27BFixed input12.7%1313.6%13
InternVideo2.5-8BFixed input12.4%1413.4%14
VideoLLaMA3-7BFixed input12.4%1413.1%16
Video-XL-2Fixed input12.4%1413.3%15
Cambrian-S-7BFixed input11.3%1712.2%17
Qwen3-VL-8BFixed input9.5%1810.2%18
VideoChat3-4BFixed input9.4%1910.2%18

Systems with the same accuracy share a rank.

For the 219 long-video items that Claude Opus 5 could not confirm at 512 frames, a human reviewer and agents verified the answers together, and we removed the items that remained uncertain. These removals are 43 of the 59 removed items above, and 3 of the 4 corrected answers are also among these long videos.

The 219 long-video itemsItems
Removed as uncertain43
Answer corrected3
Answer kept as marked173

Before the audit, Gemini 4 Argon or GPT-6-Astra already selected the marked answer on 158 of the 219 items without seeing it. Removing the uncertain items raises the accuracy of the strong systems on the long videos and leaves their order unchanged.

SystemProtocolAccuracy on the 219 itemsAccuracy on the 176 remaining items
Gemini 4 ArgonAgent54.8%68.2%
Gemini 4 ArgonFixed input46.6%56.8%
Claude Opus 5Fixed input26.5%32.4%
13 open modelsFixed input1.8–10.0%2.3–11.4%

Finally, under the same fixed input, we compared 200 items whose answer Claude Opus 5 verified against the video with 200 long-video items that it could not verify.

Fixed inputVerified itemsUnverified itemsGap
Claude Opus 5 (verifier)66.5%27.5%39.0
Gemini 4 Argon (outside construction)85.5%47.0%38.5

The gap between verified and unverified items is the same for the verifier and for a model outside construction, so verification neither favors nor penalizes the verifier.

Q1. Participants and viewing protocol of the human baseline

Eight graduate students answered the 840 items, and each of them answered 105 items. The interface plays the video muted in a browser player with seeking and replay, and there is no time limit. Each participant first answers an item without the video, from the question and options alone, and then answers it again with the video. A screenshot of the annotation interface is shown in Figure X1 (Appendix X).

Q2. Sensitivity to the model used for shortcut filtering

Thank you for this question. We ran the experiment it describes. Three replacement screens re-ran all five screening steps in place of Qwen3-VL-8B and Qwen3-VL-2B. Qwen3-VL-32B and Gemma 3 27B are stronger models and each runs every step. The InternVL3.5 screen mirrors the original one, with InternVL3.5-8B in the three shortcut steps and InternVL3.5-2B in the two 32-frame steps. Prompts, option orders, frame sampling and removal rules were unchanged. A stronger screening model keeps a partly different set of items, but the ranking of the evaluated systems is almost unchanged.

Replacement screenOptions onlyBlindSingle frame32 frames32 frames, shuffledAny stepSystem pairs that change orderSpearman ρ of the ranking
InternVL3.5-8B with InternVL3.5-2B1.0%0.7%5.0%3.1%2.6%9.6%2.3%0.99
Qwen3-VL-32B1.1%1.5%4.6%19.9%13.6%23.5%7.6%0.95
Gemma 3 27B2.6%4.2%5.1%8.9%8.8%17.3%4.1%0.96

The step columns give the share of the 840 Video-Index items that each step of the replacement screen would remove. As a measure of sensitivity, we count the pairs among the 19 systems whose order changes on the items that the screen keeps. Removing 23.5% of the items changes the order of only 7.6% of the pairs. Across the three screens the share of changed pairs is a quarter to a third of the share of removed items, and every changed pair lies between open models with fixed input. Video-Index is also largely insensitive to the model used in the shortcut steps. After the shortcut screens, the selection filters again by difficulty with the 2B model and takes the hardest remaining items, so the shortcut steps of another model remove few items and most removals come from the 32-frame steps.

The six highest-ranked systems keep their order under every screen.

SystemProtocolAll 840 itemsKept by InternVL3.5 (759)Kept by Qwen3-VL-32B (643)Kept by Gemma 3 27B (695)
Gemini 4 ArgonAgent80.4%79.6%75.7%78.1%
GPT-6-AstraAgent79.3%78.2%74.2%77.5%
Gemini 4 ArgonFixed input76.8%75.9%71.1%74.0%
Claude Fable 5.1Agent71.5%70.5%65.9%69.3%
Claude Opus 5Agent70.6%69.8%63.3%67.0%
Claude Opus 5Fixed input56.8%55.1%49.6%51.1%

On the pool, the stronger screens are as strict as the original screen and treat the source benchmarks in a similar order. Each screened a uniform random sample of 6,000 of the 269,930 multiple-choice candidates that entered the original screen.

On 6,000 pool candidatesOriginal screenQwen3-VL-32BGemma 3 27B
Candidates removed76.4%78.6%75.0%
Same decision as the original screen82.3%79.4%
Of the candidates the original removes, also removed89.9%85.6%
Of the candidates the original keeps, also kept57.8%59.2%
Pass rate per source benchmark, Spearman ρ with the original (79 benchmarks)0.830.88

The revision reports these results in Appendix W (Tables 57 to 60) and releases the per-item decisions of all four screens, so that users can also score a system on the strictest subset. The conclusions that Video-Index gives are therefore insensitive to the screening model.

Reviewer_hLcnOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates shortcut exploitation in video understanding benchmarks through a five-level attack framework applied to 115 benchmarks. Based on the audit, the authors introduce Video-Index, a curated meta-benchmark containing 840 questions from 76 sources, constructed through shortcut filtering, coverage-aware selection, and answer verification. The paper further evaluates open-source video models and tool-augmented agents.

Critical strengths

S1. Important problem. The paper addresses shortcut exploitation in video understanding benchmarks, an important issue for reliable model evaluation.

S2. Extensive empirical study. The unified audit of 115 benchmarks, supported by sensitivity analyses and multiple attack settings, provides valuable empirical evidence.

S3. Practical benchmark construction. The integration of shortcut filtering, coverage-aware selection, and reproducible benchmark generation offers a useful evaluation framework.

Critical weaknesses

W1. Reference-dependent breaking criterion. Section 2 defines a benchmark as broken when attack accuracy approaches reference-model accuracy within a 5-point tolerance. When the reference model performs near chance, even weak attacks may satisfy this criterion, as illustrated by EgoPlan-Bench.

W2. Misalignment between attacks and evaluated capabilities. Not all benchmarks require temporal reasoning, so robustness to frame shuffling does not necessarily indicate a shortcut. Appendix O shows that claim-matched evaluation increases the number of surviving benchmarks from 38 to 53.

W3. Insufficient construct validity. Resistance to predefined shortcut attacks does not establish that questions measure their intended video understanding capabilities. Independent evidence-level validation is needed to support this claim.

W4. Model-dependent selection and verification. The construction pipeline relies on specific models for difficulty filtering and answer verification (Section 5). This raises concerns about selection bias and correlated verification errors, which are not sufficiently assessed through independent validation.

W5. Limited end-to-end evaluation of on-demand construction. Section 5 demonstrates benchmark selection from structured specifications but provides limited quantitative evaluation of natural-language specification generation and constraint satisfaction.

W6. Unclear advantages over alternative selection strategies. Table 47 shows lower shortcut exploitability for Video-Index, but not consistently better Video Gain, SNR, or ranking agreement. The trade-offs between these evaluation objectives remain insufficiently explained.

W7. Limited causal evidence for temporal reasoning. The frame-shuffling results do not distinguish genuine temporal understanding from reliance on static visual cues or tasks that do not require temporal order.

W8. Unequal evaluation budgets. Table 1 compares fixed-input video models with tool-augmented agents under different evidence-access and inference conditions, limiting conclusions about the source of their performance differences.

W9. Unexplained below-chance performance. Several open models in Table 1 perform below the reported average chance accuracy of 29.7%. The potential effects of answer formatting, option bias, and item selection warrant further analysis.

Questions for authors

Q1. How do the main breaking statistics change when requiring attacks to significantly exceed chance accuracy or excluding near-chance reference models?

Q2. Given the substantial difference in Appendix O, why is claim-matched evaluation not used as the primary breaking criterion?

Q3. Can the authors provide independent human or cross-model validation of answer correctness and the capabilities required by the selected questions?

Q4. Can the authors quantify specification accuracy, constraint satisfaction, and benchmark quality across diverse natural-language requests?

9 comments

W1 / Q1. Breaking criterion with a near-chance reference

Thank you for this question. We recounted all levels under both rules that the reviewer suggests. The breaking levels are stable under both. Of the 35 breaks before video, 30 remain when the attack must be significantly above chance, and 88.8% of benchmark pairs keep their order.

RuleBenchmarksBreak before videoUnbroken
Main audit1153538
Attack significantly above chance (one-sided, 1%)1153044
Excluding the 8 near-chance references1072838

A reference near chance is rare and concerns 8 of the 115 benchmarks. In these cases the breaking level still reports what the audit measures, namely whether the video adds accuracy beyond the attack. On EgoPlan-Bench the reference reaches 29.3% with the video against 25% chance and a blind reader reaches 33%, so at the audit budget the video gives the reference no measurable advantage over the text.

W2 / Q2. Claim-matched evaluation as the primary criterion

Thank you for this question. Our audit targets video understanding benchmarks, and temporal order is what distinguishes a video from a set of images. When a score is unchanged after the frames are shuffled, it does not rely on temporal order, and in this respect the benchmark behaves like a multi-image benchmark, whatever capability it claims. The order level records this property of the score for every benchmark.

We use a single pyramid for all benchmarks as the primary criterion so that the breaking levels can be compared fairly across benchmarks. Claim-matched evaluation aligns more closely with the stated claim of each benchmark. However, most benchmarks assess several capabilities at once, so exempting attacks according to one claim does not yield a fair comparison. In our item labels (Table 23, Appendix O), the dominant capability group accounts for 65% of the items of a benchmark on average, and 72 of the 97 benchmarks outside the temporal group still contain temporal items. The claim-matched recount in Appendix O leads to the same conclusions, and its 15 additional survivors are exactly the benchmarks that break only at the order level and lie outside the temporal group.

Main capability groupBenchmarksMean share of items in the main groupBenchmarks with temporal items
Perception4563%37
Reasoning / knowledge3264%24
Spatial / physical2077%11
Temporal1859%18
All11565%90
All attacksClaim-matched attacks
Survive every level3853
Break before video3529
Break at the option level1717
Temporal group (18 benchmarks), break on options / survive6 / 106 / 10

A single criterion for all benchmarks is therefore the fair primary choice, and the claim-matched view does not change our conclusions.

W3. Construct validity of items that resist the attacks

Thank you for this comment. It contains two separate questions, namely whether an item measures its intended capability and whether an item can be answered through a shortcut.

The first question concerns the design of the source benchmarks. Every item in Video-Index is taken unchanged from its source benchmark, and the authors of that benchmark designed the item to test a capability. Whether an item measures its intended capability is therefore a property of the source design, and re-validating that design is not the goal of our work.

The second question is our motivation. An item that can be answered through a shortcut, without the video or without the evidence that it asks about, cannot reflect the capability that it was designed for, however carefully it was designed. Resistance to shortcuts is therefore a necessary condition for an item to measure its intended capability, and it is the condition that our audit tests and that Video-Index enforces.

As the reviewer requests, we also ran experiments at the level of the evidence on the items that pass every level, independently of the attacks.

The answers depend on locating the right evidence in the video. Open-source models answer from the images that Claude Opus 5 selected while answering as an agent in Claude Code, and from uniformly spaced frames at native resolution, twice as many on average. More uniform frames at higher resolution do not help, and the selected evidence raises accuracy by 8 to 11 points. This shows that an item is answered only when the specific evidence for it is found, which is what measuring a capability on video requires and what a shortcut would not need.

ModelFixed protocolUniform frames at native resolution, twice as manyImages selected by the agent
Qwen3-VL-32B21.0%21.0%32.4%
GLM-4.5V17.1%15.2%24.3%
Gemma 3 27B11.0%11.4%19.5%
Qwen3-VL-8B6.7%6.7%14.8%

The items need the video. Every system scores far higher with the video than without it, which shows that the answers are not available from the question and the options.

SystemProtocolWith the videoWithout the video
GPT-6-AstraAgent79.3%18.7%
Claude Fable 5.1Agent71.5%20.7%
Claude Opus 5Agent70.6%24.9%
Claude Opus 5Fixed input56.8%21.3%

The temporal items also depend on the order and the direction of the frames. For twelve open models, accuracy on the 210 temporal items falls by 3 to 15 points when the frames are shuffled and by 3 to 16 points when they are reversed, which shows that these items measure temporal understanding and not static cues.

The capability labels are confirmed by models outside Claude. Two models relabelled all 840 items from the question, the options and the answer, as Claude Opus 5 did, and their agreement shows that the capability of an item does not rest on the judgment of one model.

Labelling modelSame capability group as the released labelCohen's κ
Qwen3-VL-32B77.7%0.70
Gemma 3 27B67.5%0.57
At least one of the two81.8%

Resistance to the attacks is therefore not our only evidence. These experiments are independent of the attacks, and they show that items which pass every level are answered from specific visual evidence in the video. The revision reports them in Table 21 (Appendix M), Table 51 (Appendix V) and Table 62 (Appendix X).

W4 / Q3. Independent validation of selection and verification

Thank you for this question. We added independent validation for each step that relies on a specific model, namely the screening model, the answer verification, and the capability labels.

For the screening model, we re-ran all five screening steps on the 840 Video-Index items with three replacement screens, namely InternVL3.5 at the two sizes of the original screen and two stronger models that run every step. Under every replacement screen, the six highest-ranked systems keep their order and the rank agreement is at least 0.95.

Replacement screenOptions onlyBlindOne frame32 framesShuffled 32 framesAny stepRank agreement
InternVL3.5-8B and InternVL3.5-2B1.0%0.7%5.0%3.1%2.6%9.6%0.99
Qwen3-VL-32B1.1%1.5%4.6%19.9%13.6%23.5%0.95
Gemma 3 27B2.6%4.2%5.1%8.9%8.8%17.3%0.96

The table lists the share of the 840 Video-Index items that each step would remove. Rank agreement is Spearman ρ between the ranking of the 19 systems on all 840 items and on the items that the screen keeps.

For answer correctness, a human audit against the original videos removed the 59 items whose answer is in doubt and corrected 4 answers. The 19 systems keep their order on the remaining 781 items, with no pair reversed.

SystemProtocolAccuracy on 840 itemsRank on 840 itemsAccuracy on the 781 itemsRank on the 781 items
Gemini 4 ArgonAgent80.4%186.8%1
GPT-6-AstraAgent79.3%286.6%2
Gemini 4 ArgonFixed input76.8%382.2%3
Claude Fable 5.1Agent71.5%478.1%4
Claude Opus 5Agent70.6%576.3%5
Claude Opus 5Fixed input56.8%660.3%6
Qwen3-VL-32BFixed input23.0%725.0%7
InternVL3.5-30B-A3BFixed input21.4%822.7%8
Molmo2-8BFixed input19.0%920.5%9
GLM-4.5VFixed input19.0%920.0%10
InternVL3.5-8BFixed input15.2%1116.4%11
LLaVA-OneVision-2-8BFixed input14.2%1215.1%12
Gemma 3 27BFixed input12.7%1313.6%13
InternVideo2.5-8BFixed input12.4%1413.4%14
VideoLLaMA3-7BFixed input12.4%1413.1%16
Video-XL-2Fixed input12.4%1413.3%15
Cambrian-S-7BFixed input11.3%1712.2%17
Qwen3-VL-8BFixed input9.5%1810.2%18
VideoChat3-4BFixed input9.4%1910.2%18

Systems with the same accuracy share a rank. Verification by Claude Opus 5 also leaves its own score unaffected. Under the same fixed input, we compared 200 items whose answer it verified against the video with 200 long-video items that it could not verify, and the gap is the same for the verifier and for a model outside construction.

Fixed inputVerified itemsUnverified itemsGap
Claude Opus 5 (verifier)66.5%27.5%39.0
Gemini 4 Argon (outside construction)85.5%47.0%38.5

For the capabilities required by the items, two models outside Claude relabelled all 840 items from the question, the options and the answer, as Claude Opus 5 did. They agree with the released capability group on 78% and 68% of the items, and at least one of them agrees on 82% (table in W3 above).

These results address both concerns. The ranking does not depend on the specific model used for selection, and the answer key and the capability labels hold under verification that is independent of the models used in construction. The results on the screening models and on the capability labels are in the revision (Tables 57, 59 and 60 in Appendix W, Table 62 in Appendix X).

Comments on the paper

2 comments
AuthorsOpenReview ↗

Common Concerns on the Breaking Criterion, Screening Models, Answer Verification, Models in Construction, and the Agent Comparison

We sincerely thank all reviewers for their valuable feedback and constructive comments. The main motivation of this work is to build a video benchmark that faithfully reflects the video understanding ability of a model and, as model capabilities advance rapidly, still separates models of different ability. To reach this goal, we design attackers at several levels around the characteristics of video QA. They remove the items that contain a shortcut through which a model can answer correctly without truly understanding the video, and the remaining items form the Video-Index meta-benchmark. We are encouraged that all three reviewers regard shortcut exploitation in video benchmarks as an important problem for reliable evaluation, value the unified audit of 115 benchmarks under one protocol, and find Video-Index a useful and reproducible evaluation resource. Below, we summarize and address the concerns raised by more than one reviewer before providing reviewer-specific responses.

Q1. Criterion for declaring a benchmark broken (Reviewer hLcn and Reviewer HsMQ). The attack pyramid is a layered diagnostic, and we use it to describe where current benchmarks are vulnerable, not to attach a definitive label to each benchmark. In Appendix P, we change δ to 3% or 8%, and 90% and 80% of benchmark pairs keep the original order of breaking levels. In addition, we resample the audited items of every benchmark. For 86 of the 115 benchmarks, whether the benchmark breaks before or after visual input is unchanged in at least 90% of the resamples. A break is also not an artifact of a weak reference model. Of the 35 benchmarks that break before any visual input, 30 are broken by an attacker whose accuracy is significantly above chance.

Q2. Dependence on the screening models (Reviewer hLcn, Reviewer oC8v and Reviewer HsMQ). We agree that building a benchmark with specific models introduces bias. We re-ran every screening step with InternVL3.5 at the sizes of the original screen (8B and 2B) and with two stronger models, Qwen3-VL-32B and Gemma 3 27B. A stronger model removes more items, and the three replacements would remove 9.6%, 23.5% and 17.3% of Video-Index. This does not change the ranking of systems. On the items that each screen keeps, the six highest-ranked systems stay in the same order, and Spearman ρ with the ranking on all items is at least 0.95. In addition, the different screens filter the source benchmarks in a similar order. On a random sample of 6,000 candidates from the pool, the share of candidates that pass, computed per source benchmark, correlates between the original screen and the Qwen3-VL-32B and Gemma 3 27B screens at Spearman ρ = 0.83 and 0.88.

Q3. Reliability of the answer verification (Reviewer hLcn and Reviewer oC8v). Every answer key comes from its source benchmark. Claude Opus 5 only checks that the key is supported by the video and never modifies the ground truth. A human audit against the original videos removed 59 items whose ground truth is in doubt and corrected 4 answers. On the remaining 781 items, the 19 systems keep the order they have on all 840 items, with no pair reversed. In future work, we will design a reliable automated pipeline for answer verification.

Q4. Models used in construction are also evaluated (Reviewer oC8v and Reviewer HsMQ). We agree that a model that takes part in construction needs a check when it is later evaluated. Qwen3-VL, which screens the pool, is excluded from the ranking-agreement statistics. Claude Opus 5 verifies answers and has no influence on which items are ranked as hard. Under the same fixed input, we compared 200 items whose answer Claude Opus 5 verified against the video with 200 long-video items that it could not verify. Claude Opus 5 scores 66.5% on the verified and 27.5% on the unverified items, a gap of 39.0 points, and Gemini 4 Argon, a model outside the construction, scores 85.5% and 47.0%, a gap of 38.5 points. Verification therefore neither favors nor penalizes the verifier. Models that played no role in construction also follow the pattern of the other systems under the same fixed protocol. Gemini 4 Argon reaches 76.8%, and the larger open models InternVL3.5-30B-A3B, GLM-4.5V and Gemma 3 27B reach 21.4%, 19.0% and 12.7%, comparable to the open models in Table 1.

Q5. Comparison between agents and fixed-input models (Reviewer hLcn and Reviewer HsMQ). We agree that a fair comparison requires the same visual information, and we ran matched experiments with Claude Opus 5 to provide it. The agent received only the frames of the fixed protocol, and a single call without tools received the same frames at the same maximum reasoning effort. With identical frames and effort, Claude Opus 5 is still 9.0 points higher as an agent than in a single call. This remaining advantage comes from choosing what to inspect and inspecting it over several turns, not from more pixels or longer reasoning. Similarly, Gemini 4 Argon scores 80.4% as an agent and 76.8% from the fixed frames. In addition, we compare the average cost per answer of every system (Table 52, Appendix V). The agents cost 0.08 to 1.80 USD per answer and the fixed-input calls 0.12 to 0.42 USD, and the ranking does not follow cost, since the highest-scoring system, Gemini 4 Argon as an agent, costs 0.10 USD per answer.

We again appreciate the reviewers' comments and provide detailed responses below. We have also updated the PDF and highlighted all revisions in blue for clarity, and we are happy to address any additional clarifications or follow-up questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.