- Rating
- 4(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3(+1)
Read review
Paper summary
Video-IFBench evaluates how well video MLLMs follow complex instructions across 1,500 samples built from roughly 700 public videos. The benchmark defines four instruction templates (Single, Multi, Selection, and Nested) that combine 32 video task types with 39 semantic and format constraint categories. Using a semi-automatic pipeline, the authors extract global descriptions and timestamped facts with Gemini-3-Pro, generate conditional branches and evaluation checklists, and manually verify the resulting dataset. Experiments across more than 20 proprietary and open-source MLLMs using programmatic checks and an LLM judge show that models struggle when instructions stack many constraints, require fine-grained temporal grounding, or depend on video-conditioned branching.
Critical strengths
-
Separating task composition (Single and Multi instructions) from video-grounded control flow (Selection and Nested if-else structures) is a well-thought-out way to test instruction following beyond simple captioning. Forcing models to verify visual or audio conditions in the video before choosing which branch of instructions to execute tests a practical capability that existing QA benchmarks miss.
-
The evaluation covers over 20 proprietary and open-source MLLMs, including both instruct and thinking variants, and the ablations in Section 4.3 are genuinely informative. Showing that performance drops much faster with constraint count than with task count (Table 2), and that models exhibit a strong positional bias toward the first branch in Selection prompts (Table 3b), gives clear diagnostic value.
Critical weaknesses
-
Relying on Gemini-3-Pro for fact extraction and subtitle transcription introduces a potential source-model bias alongside inconsistent input setups. Gemini-3-Pro and Qwen3.5-397B-A17B-Think tie on Single TCSR (79.6 vs. 79.4), yet Gemini-3-Pro leads by a wide margin on Selection (68.5 vs. 52.1) and Nested (53.7 vs. 33.0), where branch conditions come directly from Gemini-3-Pro's extracted facts. Meanwhile, GPT-5.4 only receives 50 frames without audio on videos up to 10 minutes long, and only vision-only models receive Gemini-3-Pro timestamped subtitles.
-
Some details in the examples and judge setup raise questions about evaluation reliability. In Figure 10, Qwen3.5-397B-A17B-Think is marked as passing all 12 constraints, even though its displayed output uses the wrong title, lists 6 numbered items instead of 3, and mentions the forbidden circular object. Additionally, both LLM judges tested belong to the Qwen3.5 family, and the human agreement study in Appendix E omits the sample size and inter-annotator agreement.
Questions for authors
-
How much of Gemini-3-Pro's extracted facts and conditional branches were rewritten during human verification, and does Gemini-3-Pro's large lead on Selection and Nested tasks hold when models are compared under identical frame sampling rates and subtitle access?
-
In Appendix E, how many examples were annotated for the human agreement study, and did you test a judge outside the Qwen3.5 family on a small subset to check for same-family judge preference?
4 comments
Rebuttal to Reviewer EJFj (2)
3. Clarification and Correction of Figure 10 (W2)
"In Figure 10, Qwen3.5-397B-A17B-Think is marked as passing all 12 constraints, even though its displayed output uses the wrong title, lists 6 numbered items instead of 3, and mentions the forbidden circular object."
We sincerely appreciate you catching this, and apologize for the confusion caused by our careless mistake when preparing Figure 10. Upon checking our evaluation records, we found that we accidentally pasted Qwen3.5-397B-A17B-Think's response to another instruction on the same video (which explicitly asked for the title 'Analysis of the Figures', 6 numbered steps, and the circular object). For the actual instruction shown in Figure 10, Qwen3.5-397B-A17B-Think's true output starts with 'Colored Figures Analysis', lists 3 items without mentioning the circular object, and rightly passes all constraints under our evaluator. We have fixed Figure 10 in the revised manuscript.
4. Human Agreement Study and Cross-Family Judge Evaluation (W2 & Q2)
"In Appendix E, how many examples were annotated for the human agreement study, and did you test a judge outside the Qwen3.5 family on a small subset to check for same-family judge preference?"
In our human agreement study in Appendix E, we sampled 280 model responses across all four instruction categories (100 Single, 100 Multi, 40 Selection, and 40 Nested). Each Single and Multi response was independently evaluated by 2 annotators, and each Selection and Nested response by all 4 annotators, resulting in 720 response-level human evaluations and 2,878 constraint-level judgments in total, with an inter-annotator constraint agreement of 88.4%. We have added these details to Appendix E of the revised manuscript.
To check for potential same-family judge preference, we followed your suggestion and re-evaluated over 9k paired model responses (covering ~50k constraint judgments across 12 representative models) using GPT-5.4 as judge. On these paired responses, GPT-5.4 reaches 89.5% agreement on constraint judgments, 89.6% on task pass, and 84.0% on TISR with our main judge (Qwen3.5-397B), and yields a Spearman rank correlation of 0.993 for TCSR and 0.979 for TISR. As shown below, under GPT-5.4 as the judge, Qwen3.5-397B-A17B-Think remains the top open-source model and the relative rankings across model families remain largely consistent, supporting that our main findings are robust to the choice of judge family.
| Model | Qwen3.5-397B Judge on Paired Subset (TCSR / TISR, %) | GPT-5.4 Judge on Paired Subset (TCSR / TISR, %) |
|---|---|---|
| Gemini-3-Pro | 77.5 / 56.6 | 81.9 / 57.0 |
| Qwen3.5-397B-A17B-Think | 69.9 / 48.2 | 73.2 / 47.5 |
| Gemini-3-Flash | 70.6 / 48.6 | 71.9 / 43.6 |
| Doubao-Seed-2.0-Pro | 62.7 / 39.7 | 63.6 / 35.5 |
| Gemma-4-31B-it | 59.3 / 35.4 | 62.5 / 36.6 |
| Qwen3.5-397B-A17B-Instruct | 56.4 / 31.9 | 59.1 / 29.7 |
| Qwen3-Omni-30B-A3B-Think | 53.0 / 26.6 | 54.2 / 27.7 |
| InternVL3.5-241B-A28B-Think | 52.7 / 23.3 | 53.8 / 21.4 |
| Qwen3.5-122B-A10B-Instruct | 52.6 / 26.9 | 53.0 / 23.8 |
| InternVL3.5-241B-A28B-Instruct | 50.1 / 22.3 | 50.5 / 21.4 |
| Qwen3.5-35B-A3B-Instruct | 47.2 / 23.3 | 48.9 / 21.9 |
| Qwen3-Omni-30B-A3B-Instruct | 42.9 / 17.3 | 47.2 / 19.5 |
We hope these experiments and discussions help clarify your questions, and look forward to your feedback.
The authors resolved all of my concerns regarding Figure 10, cross-family judge agreement, and frame/subtitle parity, and added a strong ablation separating video perception from conditional routing. Raising Soundness to 3 and Rating to 4.
Dear Reviewer EJFj,
We sincerely appreciate your encouraging feedback. Your constructive suggestions helped us strengthen the evaluation and analysis of our work, and we are grateful for your support.
Rebuttal to Reviewer EJFj (1)
Dear Reviewer EJFj,
We sincerely appreciate your positive assessment of our work and your thoughtful review. We have carefully studied your comments, tried to extract all of your concerns and questions below, and conducted additional analyses in an effort to respond to each of them (also updated in the revised manuscript).
1. Concern on Potential Source-Model Bias and Human Verification (W1 & Q1a)
Using a strong model for initial annotation followed by human verification is a standard practice in recent video benchmarks (such as IF-VidCap [1], EgoSchema [2], and LongVALE [3]), as it reduces the high cost of annotating long videos from scratch and improves efficiency. As noted in Sec. 3.2 ("problematic samples are corrected or discarded"), rather than heavily rewriting flawed conditional trees from scratch, we primarily discarded over half (54.2%) of the candidate instructions whenever their extracted facts or branch conditions had any ambiguity during two-annotator verification (81.9% inter-annotator agreement), while manually correcting minor factual, timestamp, or condition wording issues in the retained samples (about 25%).
To see what drives the gap on Selection and Nested, we compared Target Branch Only (giving only the ground-truth branch task) and With GT Condition Labels (keeping the conditional branches, while directly giving the ground-truth True/False label of every condition in the prompt). On Target Branch Only, Gemini-3-Pro and Gemma-4-31B-it perform almost the same (70.5% vs. 69.2% TISR). However, under With GT Condition Labels, even though models no longer need to identify Gemini-3-Pro's extracted condition facts from the video, a large gap still appears (68.1% vs. 47.5% TISR). This suggests that handling multi-branch conditional structures plays a major role in the performance gap, beyond simply recognizing Gemini-extracted condition facts:
2. Input Settings and Comparison Under Identical Frame Sampling and Subtitles (W1 & Q1b)
We understand this concern. Because our videos are up to 10 minutes long, our main consideration was to provide as much video information as possible to each model, so that models would not fail simply because the relevant frames or speech were missing from the input. Under this principle, most models in our evaluation share a consistent 1 FPS setup, vision-only models receive timestamped subtitles because they lack an audio interface, and a few models take fewer frames due to context-length or API limits, which we view as a practical limitation of the model or its serving interface itself (Appendix F).
At the same time, following your suggestion, we re-evaluated representative models on all Selection and Nested instructions under the exact same input setting (uniform 50 frames and identical Gemini-3-Pro subtitles, without native audio). As shown below (evaluated with Qwen3.5-35B-A3B as the judge), Gemini-3-Pro still leads Qwen3.5-397B-A17B-Think and other models by a wide margin on both Selection and Nested, and Qwen3.5-397B-A17B-Think remains the strongest open-source model, consistent with our main table.
References
[1] Li et al. IF-VidCap: Can Video Caption Models Follow Instructions?. 2025.
[2] Mangalam et al. EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understanding. 2023.
[3] Geng et al. LongVALE: Vision-Audio-Language-Event Benchmark Towards Time-Aware Omni-Modal Perception of Long Videos. CVPR 2025.