acceptodds
Desk rejected at ICLR 2027

Video-IFBench: Evaluating Instruction Following of Multimodal LLMs in Video Understanding Scenarios

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Multimodal Large Language Models (MLLMs) have shown strong performance in video understanding. However, their ability to follow instructions in this domain remains underexplored. Real-world video understanding requires models not only to interpret video content correctly, but also to satisfy diverse user-specified constraints. Existing benchmarks primarily focus on task accuracy rather than instruction adherence, leaving this capability insufficiently evaluated. To address this gap, we introduce Video-IFBench, a comprehensive benchmark for evaluating instruction following in video understanding. We develop a hierarchical instruction taxonomy with four templates, covering 32 task types and 39 manually designed constraint categories spanning both semantic and format requirements. To reduce annotation cost, we build a semi-automatic data construction pipeline that combines MLLMs, programmatic processing, and rigorous human verification, resulting in 1.5K samples. We conduct a large-scale evaluation of more than 20 recent MLLMs and show that video instruction following remains challenging for current models, especially for instructions with many constraints, semantic-related constraints, or complex conditional structures that require selecting the correct branch or path based on video content. We hope our work will facilitate future research on instruction following in video understanding scenarios.

Primary area:
datasets and benchmarks
Keywords:
MLLM, VLM, Instruction Following, Video Understanding

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_EJFjOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

Video-IFBench evaluates how well video MLLMs follow complex instructions across 1,500 samples built from roughly 700 public videos. The benchmark defines four instruction templates (Single, Multi, Selection, and Nested) that combine 32 video task types with 39 semantic and format constraint categories. Using a semi-automatic pipeline, the authors extract global descriptions and timestamped facts with Gemini-3-Pro, generate conditional branches and evaluation checklists, and manually verify the resulting dataset. Experiments across more than 20 proprietary and open-source MLLMs using programmatic checks and an LLM judge show that models struggle when instructions stack many constraints, require fine-grained temporal grounding, or depend on video-conditioned branching.

Critical strengths
  1. Separating task composition (Single and Multi instructions) from video-grounded control flow (Selection and Nested if-else structures) is a well-thought-out way to test instruction following beyond simple captioning. Forcing models to verify visual or audio conditions in the video before choosing which branch of instructions to execute tests a practical capability that existing QA benchmarks miss.

  2. The evaluation covers over 20 proprietary and open-source MLLMs, including both instruct and thinking variants, and the ablations in Section 4.3 are genuinely informative. Showing that performance drops much faster with constraint count than with task count (Table 2), and that models exhibit a strong positional bias toward the first branch in Selection prompts (Table 3b), gives clear diagnostic value.

Critical weaknesses
  1. Relying on Gemini-3-Pro for fact extraction and subtitle transcription introduces a potential source-model bias alongside inconsistent input setups. Gemini-3-Pro and Qwen3.5-397B-A17B-Think tie on Single TCSR (79.6 vs. 79.4), yet Gemini-3-Pro leads by a wide margin on Selection (68.5 vs. 52.1) and Nested (53.7 vs. 33.0), where branch conditions come directly from Gemini-3-Pro's extracted facts. Meanwhile, GPT-5.4 only receives 50 frames without audio on videos up to 10 minutes long, and only vision-only models receive Gemini-3-Pro timestamped subtitles.

  2. Some details in the examples and judge setup raise questions about evaluation reliability. In Figure 10, Qwen3.5-397B-A17B-Think is marked as passing all 12 constraints, even though its displayed output uses the wrong title, lists 6 numbered items instead of 3, and mentions the forbidden circular object. Additionally, both LLM judges tested belong to the Qwen3.5 family, and the human agreement study in Appendix E omits the sample size and inter-annotator agreement.

Questions for authors
  1. How much of Gemini-3-Pro's extracted facts and conditional branches were rewritten during human verification, and does Gemini-3-Pro's large lead on Selection and Nested tasks hold when models are compared under identical frame sampling rates and subtitle access?

  2. In Appendix E, how many examples were annotated for the human agreement study, and did you test a judge outside the Qwen3.5 family on a small subset to check for same-family judge preference?

4 comments
AuthorsOpenReview ↗

Rebuttal to Reviewer EJFj (1)

Dear Reviewer EJFj,

We sincerely appreciate your positive assessment of our work and your thoughtful review. We have carefully studied your comments, tried to extract all of your concerns and questions below, and conducted additional analyses in an effort to respond to each of them (also updated in the revised manuscript).

1. Concern on Potential Source-Model Bias and Human Verification (W1 & Q1a)

"Relying on Gemini-3-Pro for fact extraction and subtitle transcription introduces a potential source-model bias... How much of Gemini-3-Pro's extracted facts and conditional branches were rewritten during human verification?"

Using a strong model for initial annotation followed by human verification is a standard practice in recent video benchmarks (such as IF-VidCap [1], EgoSchema [2], and LongVALE [3]), as it reduces the high cost of annotating long videos from scratch and improves efficiency. As noted in Sec. 3.2 ("problematic samples are corrected or discarded"), rather than heavily rewriting flawed conditional trees from scratch, we primarily discarded over half (54.2%) of the candidate instructions whenever their extracted facts or branch conditions had any ambiguity during two-annotator verification (81.9% inter-annotator agreement), while manually correcting minor factual, timestamp, or condition wording issues in the retained samples (about 25%).

To see what drives the gap on Selection and Nested, we compared Target Branch Only (giving only the ground-truth branch task) and With GT Condition Labels (keeping the conditional branches, while directly giving the ground-truth True/False label of every condition in the prompt). On Target Branch Only, Gemini-3-Pro and Gemma-4-31B-it perform almost the same (70.5% vs. 69.2% TISR). However, under With GT Condition Labels, even though models no longer need to identify Gemini-3-Pro's extracted condition facts from the video, a large gap still appears (68.1% vs. 47.5% TISR). This suggests that handling multi-branch conditional structures plays a major role in the performance gap, beyond simply recognizing Gemini-extracted condition facts:

ModelOriginal Conditional Prompt (TISR, %)With GT Condition Labels (TISR, %)Target Branch Only (TISR, %)
Gemini-3-Pro49.168.170.5
Gemma-4-31B-it23.847.569.2

2. Input Settings and Comparison Under Identical Frame Sampling and Subtitles (W1 & Q1b)

"Meanwhile, GPT-5.4 only receives 50 frames without audio on videos up to 10 minutes long, and only vision-only models receive Gemini-3-Pro timestamped subtitles... does Gemini-3-Pro's large lead on Selection and Nested tasks hold when models are compared under identical frame sampling rates and subtitle access?"

We understand this concern. Because our videos are up to 10 minutes long, our main consideration was to provide as much video information as possible to each model, so that models would not fail simply because the relevant frames or speech were missing from the input. Under this principle, most models in our evaluation share a consistent 1 FPS setup, vision-only models receive timestamped subtitles because they lack an audio interface, and a few models take fewer frames due to context-length or API limits, which we view as a practical limitation of the model or its serving interface itself (Appendix F).

At the same time, following your suggestion, we re-evaluated representative models on all Selection and Nested instructions under the exact same input setting (uniform 50 frames and identical Gemini-3-Pro subtitles, without native audio). As shown below (evaluated with Qwen3.5-35B-A3B as the judge), Gemini-3-Pro still leads Qwen3.5-397B-A17B-Think and other models by a wide margin on both Selection and Nested, and Qwen3.5-397B-A17B-Think remains the strongest open-source model, consistent with our main table.

Model (Uniform 50 Frames + Subtitles, No Audio)Selection (TCSR / TISR, %)Nested (TCSR / TISR, %)Overall Conditional (TCSR / TISR, %)
Gemini-3-Pro61.3 / 52.543.1 / 38.252.2 / 45.3
Qwen3.5-397B-A17B-Think46.2 / 40.827.0 / 23.036.6 / 31.9
Gemma-4-31B-it33.9 / 29.918.5 / 13.826.2 / 21.9
Qwen3.5-397B-A17B-Instruct24.6 / 19.917.2 / 13.120.9 / 16.5
InternVL3.5-30B-A3B-Think28.8 / 19.017.8 / 10.223.3 / 14.6
GPT-5.422.9 / 17.612.2 / 9.217.6 / 13.4

References

[1] Li et al. IF-VidCap: Can Video Caption Models Follow Instructions?. 2025.

[2] Mangalam et al. EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understanding. 2023.

[3] Geng et al. LongVALE: Vision-Audio-Language-Event Benchmark Towards Time-Aware Omni-Modal Perception of Long Videos. CVPR 2025.

AuthorsOpenReview ↗

Rebuttal to Reviewer EJFj (2)

3. Clarification and Correction of Figure 10 (W2)

"In Figure 10, Qwen3.5-397B-A17B-Think is marked as passing all 12 constraints, even though its displayed output uses the wrong title, lists 6 numbered items instead of 3, and mentions the forbidden circular object."

We sincerely appreciate you catching this, and apologize for the confusion caused by our careless mistake when preparing Figure 10. Upon checking our evaluation records, we found that we accidentally pasted Qwen3.5-397B-A17B-Think's response to another instruction on the same video (which explicitly asked for the title 'Analysis of the Figures', 6 numbered steps, and the circular object). For the actual instruction shown in Figure 10, Qwen3.5-397B-A17B-Think's true output starts with 'Colored Figures Analysis', lists 3 items without mentioning the circular object, and rightly passes all constraints under our evaluator. We have fixed Figure 10 in the revised manuscript.

4. Human Agreement Study and Cross-Family Judge Evaluation (W2 & Q2)

"In Appendix E, how many examples were annotated for the human agreement study, and did you test a judge outside the Qwen3.5 family on a small subset to check for same-family judge preference?"

In our human agreement study in Appendix E, we sampled 280 model responses across all four instruction categories (100 Single, 100 Multi, 40 Selection, and 40 Nested). Each Single and Multi response was independently evaluated by 2 annotators, and each Selection and Nested response by all 4 annotators, resulting in 720 response-level human evaluations and 2,878 constraint-level judgments in total, with an inter-annotator constraint agreement of 88.4%. We have added these details to Appendix E of the revised manuscript.

To check for potential same-family judge preference, we followed your suggestion and re-evaluated over 9k paired model responses (covering ~50k constraint judgments across 12 representative models) using GPT-5.4 as judge. On these paired responses, GPT-5.4 reaches 89.5% agreement on constraint judgments, 89.6% on task pass, and 84.0% on TISR with our main judge (Qwen3.5-397B), and yields a Spearman rank correlation of 0.993 for TCSR and 0.979 for TISR. As shown below, under GPT-5.4 as the judge, Qwen3.5-397B-A17B-Think remains the top open-source model and the relative rankings across model families remain largely consistent, supporting that our main findings are robust to the choice of judge family.

ModelQwen3.5-397B Judge on Paired Subset (TCSR / TISR, %)GPT-5.4 Judge on Paired Subset (TCSR / TISR, %)
Gemini-3-Pro77.5 / 56.681.9 / 57.0
Qwen3.5-397B-A17B-Think69.9 / 48.273.2 / 47.5
Gemini-3-Flash70.6 / 48.671.9 / 43.6
Doubao-Seed-2.0-Pro62.7 / 39.763.6 / 35.5
Gemma-4-31B-it59.3 / 35.462.5 / 36.6
Qwen3.5-397B-A17B-Instruct56.4 / 31.959.1 / 29.7
Qwen3-Omni-30B-A3B-Think53.0 / 26.654.2 / 27.7
InternVL3.5-241B-A28B-Think52.7 / 23.353.8 / 21.4
Qwen3.5-122B-A10B-Instruct52.6 / 26.953.0 / 23.8
InternVL3.5-241B-A28B-Instruct50.1 / 22.350.5 / 21.4
Qwen3.5-35B-A3B-Instruct47.2 / 23.348.9 / 21.9
Qwen3-Omni-30B-A3B-Instruct42.9 / 17.347.2 / 19.5

We hope these experiments and discussions help clarify your questions, and look forward to your feedback.

The authors resolved all of my concerns regarding Figure 10, cross-family judge agreement, and frame/subtitle parity, and added a strong ablation separating video perception from conditional routing. Raising Soundness to 3 and Rating to 4.

Dear Reviewer EJFj,

We sincerely appreciate your encouraging feedback. Your constructive suggestions helped us strengthen the evaluation and analysis of our work, and we are grateful for your support.

Reviewer_3HsoOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces Video-IFBench, a benchmark of 1.5K samples designed to evaluate how well models follow instructions when understanding videos. It covers single task, multi task, selection, and nested instructions, with a range of semantic and formatting constraints. The paper evaluates more than 20 models and find that even strong models struggle with complex instructions.

Critical strengths

This paper addresses a meaningful gap in video evaluation. I like that the authors has included conditional and nested instructions, which tests capabilities beyond conventional video question answering.

The evaluation covers a broad range of models and the analysis of constraints count, branch position, and nesting depth help explain where models struggle.

The human verification process and the study of agreement between automatic and human evaluators also add credibility to the benchmark.

Critical weaknesses

My main concern is the paper’s novelty. IF-VidCap already examines video instruction following with content and formatting constraints. Video-IFBench expands the scope to more tasks and more complex instruction structures, but the methodological contribution still feels incremental.

An important issue is that the task-gated metrics do not clearly separate instruction following from video understanding. A model might fail because it misses an event in the video, rather than because it cannot follow the instruction. This makes it difficult to determine exactly what the benchmark results tell us about instruction-following ability. An experiment that provides models with ground-truth video descriptions would help disentangle these two sources of error and make the conclusions more convincing.

Some of the deeply nested instructions also feel overly artificial, and it is unclear how closely they reflect realistic user requests. The human agreement study is useful but its sample size is not clearly reported. Finally, differences in video input settings across models make direct comparisons harder to interpret.

Questions for authors
  1. Can you separate video comprehension errors from instruction-following errors, perhaps by evaluating models with ground truth video descriptions?

  2. How many samples were used in the human agreement study, and how sensitive are the model rankings to judge disagreement?

  3. How representative are the deepest nested instructions of requests users would make in practice?

6 comments
AuthorsOpenReview ↗

Rebuttal to Reviewer 3Hso (1)

Dear Reviewer 3Hso,

Thank you for taking the time to review our paper and for your constructive feedback. We have carefully reflected on your comments, tried to summarize all of your concerns below, and conducted additional experiments and analyses in an effort to respond to each of them (also updated in the revised manuscript).

1. Concern on Novelty and Methodological Contribution Beyond IF-VidCap (W1)

"My main concern is the paper’s novelty. IF-VidCap already examines video instruction following with content and formatting constraints. Video-IFBench expands the scope to more tasks and more complex instruction structures, but the methodological contribution still feels incremental."

Thank you for raising this point. However, for a benchmark work, we believe the evaluation methodology should prioritize reliability and diagnostic clarity, and the core novelty lies in evolving what capabilities are evaluated to keep pace with real-world video understanding needs.

On the methodological side, rather than complicating the scoring mechanism for its own sake, our goal is to make evaluation on long videos and conditional instructions reliable and verifiable. We combine task-level gating, deterministic programmatic verifiers for format rules, and context-grounded checklist judging, paired with a multi-stage construction pipeline (Sec. 3.2) that tracks entities across video chunks and constructs verified conditional paths.

On the benchmark novelty side, while IF-VidCap [1] is a valuable starting point, it focuses on single-task captioning over 350 short visual clips (averaging 20.5s, ~2 hours in total) without systematic audio-visual evaluation. Because its prompts center on short-clip captioning, it cannot diagnose which specific video capability causes an error, or evaluate multi-hop perception where a model must link multiple video cues across segments. Video-IFBench doubles the video coverage to 700+ videos (~49 hours, up to 10 minutes per video) with audio-visual inputs, organizes 32 tasks into 6 categories alongside 39 constraints for fine-grained capability diagnosis, and introduces Multi, Selection, and Nested structures to test multi-hop perception systematically.

Crucially, this reveals bottlenecks that single-task captioning benchmarks cannot see: models that already score well on single-step instructions suffer severe drops when required to link multiple video cues, where simply scaling model parameters is far less effective than explicit step-by-step reasoning. In fact, our additional analysis (Appendix A.4 of the revised manuscript) shows that even on cases where a model judges all candidate conditions correctly in isolation and satisfies the target-branch instruction completely, full conditional execution still fails in 57.0% of cases for Gemma-4-31B-it and 26.9% for Gemini-3-Pro.

References

[1] Li et al. IF-VidCap: Can Video Caption Models Follow Instructions?. 2025.

AuthorsOpenReview ↗

Rebuttal to Reviewer 3Hso (2)

2. Clarifying the Task-Gated Metric and Separating Video Comprehension from Instruction Following (W2 & Q1)

"An important issue is that the task-gated metrics do not clearly separate instruction following from video understanding... Can you separate video comprehension errors from instruction-following errors, perhaps by evaluating models with ground truth video descriptions?"

First, we would like to clarify the role of the task gate (tit_i). As defined in Sec. 3.3, tit_i checks whether the response addresses all required tasks in the active instruction (task execution), rather than whether all video details are factually correct. The purpose of tit_i is simply to prevent a model from receiving credit for formatting constraints when it ignores the requested task altogether.

In conditional instructions (Selection and Nested), requiring the model to ground conditions in video events and select the corresponding branch (making ti=0t_i=0 if it chooses the wrong branch) is a core part of our benchmark design. At the same time, to explicitly separate how much of the error comes from missing video events versus instruction following itself, we followed your suggestion and evaluated all conditional instructions on the same videos under four settings:

  • Original Conditional Prompt: The model receives the raw video and the full If / Else-if / Else instruction.
  • With GT Video Description: We additionally provide our human-verified ground-truth video description in the prompt.
  • With GT Condition Labels: Because a general video description may still omit specific negative conditions in false branches, we also test explicitly giving the ground-truth True/False status of every candidate condition in the prompt.
  • Target Branch Only: We remove all conditional branches and directly give the model the ground-truth branch task and constraints.
ModelOriginal Conditional Prompt (TISR, %)With GT Video Description (TISR, %)With GT Condition Labels (TISR, %)Target Branch Only (TISR, %)
Gemini-3-Pro49.159.168.170.5
Gemma-4-31B-it23.835.447.569.2

These results help separate the two error sources:

(a) Contribution of video comprehension errors: Adding the ground-truth video description improves TISR by +10.0 to +11.6 points, and providing exact condition truth labels improves TISR by +19.0 to +23.7 points. This confirms that missing fine-grained video events does account for a meaningful share of end-to-end failures.

(b) Instruction following remains a major bottleneck: Even when exact condition truth labels are provided in the prompt (With GT Condition Labels), models still fail on 31.9% to 52.5% of the instructions (and 40.9% to 64.6% with ground-truth descriptions). In particular, Gemma-4-31B-it with ground-truth condition labels still lags far behind Target Branch Only (47.5% vs. 69.2%), showing that models often fail to select and execute the right branch even when condition truth values are explicitly given. Moreover, even in Target Branch Only, models still fail in 29.5% to 30.8% of cases. Thus, even after providing ground-truth video information, our core conclusion that complex instruction following is a major bottleneck remains unchanged.

AuthorsOpenReview ↗

Rebuttal to Reviewer 3Hso (3)

3. Sample Size in Human Agreement Study and Ranking Sensitivity to Judge Disagreement (W3 & Q2)

"The human agreement study is useful but its sample size is not clearly reported... How many samples were used in the human agreement study, and how sensitive are the model rankings to judge disagreement?"

Sample size in the human agreement study: In Appendix E, we sampled 280 model responses across all four instruction categories (100 Single, 100 Multi, 40 Selection, and 40 Nested). Each Single and Multi response was independently evaluated by 2 annotators, and each Selection and Nested response was evaluated by all 4 annotators, resulting in 720 response-level human evaluations and 2,878 constraint-level judgments in total. We have added these details to Appendix E of the revised manuscript.

Sensitivity of model rankings to judge disagreement: On the human-evaluated subset (covering 5 representative models with 56 responses each), the model ranking under human scores is identical to the automatic evaluator. To test ranking stability at a larger scale, we further compared our main judge (Qwen3.5-397B, Table 1), the smaller judge (Qwen3.5-35B, Table 4), and a cross-family judge (GPT-5.4, evaluated on over 9k paired model responses across 12 representative models). GPT-5.4 reaches 89.5% constraint agreement, 89.6% task-pass agreement, and 84.0% TISR agreement with Qwen3.5-397B, and the resulting model rankings remain highly consistent, with a Spearman rank correlation of 0.993 for TCSR and 0.979 for TISR (and 0.994 / 0.989 across all 38 models between Qwen3.5-397B and Qwen3.5-35B). This shows that relative model rankings are very stable across different evaluators.

4. Practical Relevance of Deeply Nested Instructions (W3 & Q3)

"Some of the deeply nested instructions also feel overly artificial, and it is unclear how closely they reflect realistic user requests... How representative are the deepest nested instructions of requests users would make in practice?"

The core goal of our Nested instructions is to evaluate conditional video understanding: different video contents require different outputs. This has clear practical value in real-world scenarios such as video monitoring and anomaly detection. In these tasks, users define different output rules for different anomaly events. By formulating these rules as explicit conditional trees, we keep the ground-truth path unambiguous for automatic verification and strike a good balance between practical relevance and reliable evaluation.

In our dataset, most Nested samples actually have shallow-to-moderate depths (depths 2 to 5 in Fig. 3b). For the deeper ones, user instructions in real applications can take any form, so we should not ignore complex instructions just because they seem less common. Representative benchmarks such as IFBench[2] follow a similar idea and include rare but verifiable instructions (for example, requiring every word in the response to contain a consonant cluster) to test model limits. Similarly, including deeper trees (depths 8 to 14) helps us test the upper limits of strong models (in Fig. 6, Gemini-3-Pro still reaches 55.6% while Qwen3.5-397B-Think drops from 40.5% to 22.2%).

5. Differences in Video Input Settings Across Models (W3)

"Finally, differences in video input settings across models make direct comparisons harder to interpret."

We understand this concern. Because our videos are up to 10 minutes long, our main consideration was to provide as much video information as possible to each model, so that models would not fail simply because the relevant frames were missing from the input. Under this principle, most models in our evaluation actually share a consistent 1 FPS setup. Only a few models take fewer frames due to context-length or API limits, and in long-video scenarios, we view this input constraint as a practical limitation of the model or its serving interface (as discussed in Appendix F). In addition, when we re-evaluate representative models under the same uniform 50-frame and subtitle input, the relative model rankings remain consistent with Table 1:

Model (Uniform 50 Frames + Subtitles, No Audio)Selection (TCSR / TISR, %)Nested (TCSR / TISR, %)Overall Conditional (TCSR / TISR, %)
Gemini-3-Pro61.3 / 52.543.1 / 38.252.2 / 45.3
Qwen3.5-397B-A17B-Think46.2 / 40.827.0 / 23.036.6 / 31.9
Gemma-4-31B-it33.9 / 29.918.5 / 13.826.2 / 21.9
Qwen3.5-397B-A17B-Instruct24.6 / 19.917.2 / 13.120.9 / 16.5
InternVL3.5-30B-A3B-Think28.8 / 19.017.8 / 10.223.3 / 14.6
GPT-5.422.9 / 17.612.2 / 9.217.6 / 13.4

We hope these experiments and discussions help clarify your questions, and look forward to your feedback and further discussion.

References

[2] Pyatkin et al. Generalizing Verifiable Instruction Following. 2025.

Reviewer_G9hjOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Video-IFBench evaluates instruction following grounded in video content. It contains approximately 1.5K examples spanning 32 tasks and 39 constraint types, organized into Single, Multi, Selection, and Nested instructions. A hybrid protocol measures task coverage and constraint satisfaction using programmatic checks and model-based judgments. The experiments compare multimodal models and analyze sensitivity to constraint load, conditional depth, branch count, and branch position.

Critical strengths

The benchmark addresses a practical capability: answering correctly is insufficient if the model chooses the wrong conditional branch or violates the user's constraints. The taxonomy and task/constraint annotations make this capability inspectable (Sec. 3).

The branch-position experiment is a useful control. Keeping the branch set fixed while moving the correct branch avoids some confounds of comparing unrelated difficulty groups. In Table 3, Qwen3.5-397B-A17B-Think drops from 76.5 to 52.9 task pass when the correct branch moves from position 1 to 3.

Human verification, human/judge agreement checks, and an alternative-judge analysis provide support for the broad reported trends.

Critical weaknesses

Limited demonstrated advance beyond extending an existing evaluation paradigm. Constraint-aware video evaluation already includes instruction compliance alongside description quality, as illustrated by IF-VidCap [1]. Video-IFBench adds meaningful conditional and multi-task coverage; I do not regard it as a duplicate. However, the strongest demonstrated increment remains a broader taxonomy plus empirical fragility under more complex instruction structures. Table 3 is informative, but the paper does not yet make a sufficiently compelling case that this extension yields substantial new knowledge about video-grounded execution, beyond documenting constraint and position sensitivity. This is a judgment about the significance of the supported increment, not a requirement to propose a new algorithm.

The main analyses support end-to-end fragility more strongly than a distinct instruction-following bottleneck. The task-gated TCSR/TISR definitions (p. 6, Eq. 1) appropriately penalize failures anywhere in the joint task, but perception errors, branch selection, and constraint execution can all reduce these scores. Moreover, all-constraints-passed rates become stricter as constraints are added, and the instruction categories use different task/constraint distributions. Consequently, comparisons across those categories do not isolate why performance changes. The controlled branch experiment partially addresses this issue, but it does not disentangle all these factors. Since the paper's contribution is empirical characterization, the resulting limit on interpretation materially reduces the strength of its scientific advance.

Reference:

Questions for authors
  1. Using the existing annotations/results, can the authors separate incorrect video-condition judgments, incorrect path selection despite a correct condition judgment, and failures to execute the selected instruction? Which headline conclusions remain after that separation?

  2. What is the most important finding specific to video-grounded conditional execution that is not adequately captured by existing constraint-aware video evaluation? Please connect that claim to the current controlled experiments rather than only to the number of tasks or constraints.

4 comments
AuthorsOpenReview ↗

Rebuttal to Reviewer G9hj (1)

Dear Reviewer G9hj,

Thank you for taking the time to review our paper and for your insightful feedback. We have carefully studied your review, tried to extract all of your concerns and questions below, and conducted additional experiments and analyses in an effort to respond to each of them (also updated in the revised manuscript).

1. Separating Condition Judgment, Path Selection, and Instruction Execution, and Remaining Headline Conclusions (W2 & Q1)

"Can the authors separate incorrect video-condition judgments, incorrect path selection despite a correct condition judgment, and failures to execute the selected instruction? Which headline conclusions remain after that separation?"

To probe how each factor contributes to overall performance, we evaluated the conditional instructions (Selection and Nested) on the same videos under four settings: (1) Isolated Condition Check, turning each candidate video condition into a separate Yes/No question; (2) Original Conditional Prompt, the standard evaluation with the video and full If / Else-if / Else instruction; (3) With GT Condition Labels, keeping the full If / Else-if / Else instruction while explicitly telling the model in the prompt which conditions are True and False; and (4) Target Branch Only, removing If / Else-if / Else branches and directly giving the ground-truth branch instruction.

ModelIsolated Condition Check Acc. (%)Original Conditional Prompt (TISR, %)With GT Condition Labels (TISR, %)Target Branch Only (TISR, %)
Gemini-3-Pro88.149.168.170.5
Gemma-4-31B-it84.123.847.569.2

From these results, we can observe the role of each stage:

  • (a) Incorrect video-condition judgments: Giving the ground-truth True/False labels for each condition (With GT Condition Labels) improves TISR by +19.0 to +23.7 points over the Original Conditional Prompt, showing that misjudging video conditions is a major source of error.
  • (b) Wrong path selection despite correct condition judgments: Even when we explicitly tell the model which conditions are True and False, Gemma-4-31B-it is still much lower than when given the Target Branch Only (47.5% vs. 69.2%, whereas Gemini-3-Pro reaches 68.1% vs. 70.5%). This shows that even when condition judgments are given, models often still fail to select and execute the right branch.
  • (c) Failures to execute the selected instruction: Even when given the target branch instruction directly (Target Branch Only), models only reach 69.2%–70.5% TISR: while format constraints are mostly satisfied (93.1%–94.5% CSR), models still frequently fail on video-grounded semantic constraints (68.2%–72.9% CSR).

Separating these stages confirms that our main conclusions continue to hold: (a) conditional structures remain a major bottleneck even when condition truth labels are provided; (b) semantic constraints remain much harder than format constraints even when branch selection is bypassed; (c) strong video perception does not automatically translate into faithful instruction execution (Gemma-4-31B-it is close to Gemini-3-Pro on Isolated Condition Check and Target Branch Only, yet drops to less than half of Gemini-3-Pro on the Original Conditional Prompt).

AuthorsOpenReview ↗

Rebuttal to Reviewer G9hj (2)

2. Stricter All-Constraints-Passed Rates and Cross-Category Comparisons (W2)

"Moreover, all-constraints-passed rates become stricter as constraints are added, and the instruction categories use different task/constraint distributions. Consequently, comparisons across those categories do not isolate why performance changes."

In real-world scenarios, users typically expect all requirements in an instruction to be satisfied for a response. This is why we report TISR alongside average per-constraint TCSR, consistent with prior benchmarks such as IFEval [2], FollowBench [3], CFBench [4], MM-IFEngine [5], and IF-VidCap [1]. Regarding the four instruction categories, they were not designed to be directly compared against one another to isolate why performance changes. Instead, each category targets a different capability: Single and Multi focus on constraint satisfaction and multi-task coverage, while Selection and Nested focus on conditional routing (Task Pass in Sec. 3.3 and 4.3 specifically checks whether models follow the right branch). For this reason, we analyze each factor within its own category in Sec. 4.3 rather than attributing cross-category differences to a single cause.

Moreover, our intervention experiments in Point 1 compare Original Conditional Prompt with Target Branch Only on the same videos, target tasks, and evaluation checklists. Thus, the observed gap does not arise from comparing different task distributions or requiring more scored constraints to pass.

References

[1] Li et al. IF-VidCap: Can Video Caption Models Follow Instructions?. 2025.

[2] Zhou et al. Instruction-Following Evaluation for Large Language Models. 2023.

[3] Jiang et al. FollowBench: A Multi-level Fine-grained Constraints Following Benchmark for Large Language Models. 2024.

[4] Zhang et al. CFBench: A Comprehensive Constraints-Following Benchmark for LLMs. 2024.

[5] Ding et al. MM-IFEngine: Towards Multimodal Instruction Following. 2025.

AuthorsOpenReview ↗

Rebuttal to Reviewer G9hj (3)

3. Key Differences from IF-VidCap and Their Importance (W1)

"Limited demonstrated advance beyond extending an existing evaluation paradigm. Constraint-aware video evaluation already includes instruction compliance alongside description quality, as illustrated by IF-VidCap [1]."

For a benchmark work, evolving the evaluation to keep pace with key trends in video understanding is essential. While IF-VidCap [1] is a valuable starting point, it focuses on single-task captioning over 350 short visual clips (averaging 20.5s, up to 60s) without audio evaluation. Video-IFBench extends this to 700+ videos up to 10 minutes (~49 hours) with audio-visual inputs, and organizes 32 tasks into 6 categories alongside 39 constraints so that users can directly diagnose model weaknesses across specific tasks and constraints (Table 8 and Fig. 4).

Furthermore, in real-world scenarios such as video monitoring, users often define multiple anomaly conditions in a single prompt and require different outputs depending on what happens in the video. Because IF-VidCap only tests single-task descriptions, it cannot evaluate this kind of multi-hop perception across time. We therefore introduce Multi, Selection, and Nested instructions to test whether models can check multiple video cues step by step and trigger the right output for each situation.

4. Core Findings Specific to Video-Grounded Conditional Execution (W1 & Q2)

"What is the most important finding specific to video-grounded conditional execution that is not adequately captured by existing constraint-aware video evaluation? Please connect that claim to the current controlled experiments rather than only to the number of tasks or constraints."

Unlike prior benchmarks where the target task is fixed in advance, conditional execution requires models to use video evidence to decide which task applies. Our controlled experiments reveal three core findings:

  • (1) Strong video perception and single-task IF do not automatically combine into reliable conditional execution: Even on the exact subset of cases where a model passes both Isolated Condition Check (100%) and Target Branch Only (100%) completely, its pass rate drops sharply when given the full Original Conditional Prompt (failing on 26.9% and 57.0% of cases, with 71.8%–72.2% of those failures executing another branch's task instead):
ModelIsolated Condition Check Pass (%)Target Branch Only (TISR, %)Original Conditional Prompt (TISR, %)
Gemini-3-Pro100.0100.073.1
Gemma-4-31B-it100.0100.043.0
  • (2) The key bottleneck in conditional routing is rejecting near-miss false conditions rather than recognizing true ones: Conditional execution critically depends on ruling out false branches that share partial visual overlap with the video. For example, Gemma-4-31B-it is nearly as good as Gemini-3-Pro at recognizing true conditions (84.5% vs. 85.0%), yet it misjudges false conditions as true more than twice as often (16.4% vs. 7.4%). This explains Table 3a and 3b: adding more false branches or placing them ahead of the true branch introduces more partially matching visual traps that models fail to reject.
  • (3) Scaling model size is not enough for conditional execution, and step-by-step verification is critical: In Table 1, Qwen3.5-397B-Instruct matches Qwen3.5-9B-Think on Single and Multi instructions, but falls far behind on Selection and Nested (18.9 / 7.9 vs. 32.0 / 11.9), and on the same 397B backbone in Table 3b, 397B-Instruct is much more sensitive to branch position than 397B-Think (dropping from 70.6% to 29.4%, vs. 76.5% to 52.9%). This matches our observation from the human study: even humans easily misjudge conditions when watching a video only once, and they often spend up to an hour repeatedly rewatching and checking every branch on a 10-minute video. Thinking models partially mimic this process through explicit reasoning.

We hope these experiments and discussions help clarify your questions, and look forward to your feedback.

References

[1] Li et al. IF-VidCap: Can Video Caption Models Follow Instructions?. 2025.

Dear Reviewer G9hj,

Thank you again for your time and thoughtful review. We wanted to gently follow up to see if our response and additional experiments have addressed your concerns, and we would be happy to discuss any further questions you may have.

Comments on the paper

2 comments

Dear AC and Reviewers,

We sincerely thank you for your time and effort in evaluating our paper. To facilitate the discussion phase, we summarize (1) our paper and core contributions, (2) the key strengths and main concerns raised by the reviewers, and (3) how we addressed each concern (all updates have been incorporated into the revised manuscript).

1. Summary of Our Paper and Contributions

Video-IFBench evaluates whether video MLLMs can faithfully follow complex user instructions while understanding videos. We construct 1,465 manually verified instructions across 700+ videos (~49 hours, up to 10 minutes), spanning 32 fine-grained video understanding tasks in 6 categories, 39 semantic and format constraints, and 4 instruction templates (Single, Multi, Selection, and Nested). Using a hybrid evaluation protocol that combines task-gated LLM-as-Judge with deterministic programmatic verification across 20+ proprietary and open-source MLLMs, we reveal critical bottlenecks in constraint scaling, semantic constraint adherence, and video-grounded conditional routing.

2. Key Strengths Highlighted by Reviewers

  • Reviewer EJFj: Commended separating task composition (Single and Multi) from video-grounded control flow (Selection and Nested) to test a practical capability missed by existing QA benchmarks; praised our broad evaluation of 20+ MLLMs (including Instruct and Thinking variants) and informative ablations on constraint count and branch-position bias; and noted after our rebuttal that "the authors resolved all of my concerns regarding Figure 10, cross-family judge agreement, and frame/subtitle parity, and added a strong ablation separating video perception from conditional routing."
  • Reviewer 3Hso: Recognized that our work addresses a meaningful gap in video evaluation beyond conventional video QA by including conditional and nested instructions, commended our broad model coverage and diagnostic analyses (constraint count, branch position, and nesting depth), and noted that our human verification and human-judge agreement study add credibility to the benchmark.
  • Reviewer G9hj: Highlighted that our benchmark addresses a practical capability where answering correctly is insufficient if the model chooses the wrong conditional branch or violates user constraints, and praised the inspectability of our task/constraint taxonomy, the controlled branch-position experiment (Table 3), and our human and alternative-judge verification analyses.

3. Summary of Main Concerns Raised by Reviewers

We have carefully reflected on the constructive concerns and questions raised by the reviewers:

  1. Disentangling error sources and potential source-model bias (G9hj W2/Q1, 3Hso W2/Q1, EJFj W1/Q1a): Separating video comprehension and condition judgment errors from path selection and constraint execution errors, and examining whether using Gemini-3-Pro in data construction introduces source-model bias on conditional tasks.
  2. Novelty and core scientific findings beyond IF-VidCap (G9hj W1/Q2, 3Hso W1): Clarifying the methodological and benchmark advances beyond prior constraint-aware video captioning (IF-VidCap), and connecting our controlled experiments to substantial new findings specific to video-grounded conditional execution.
  3. Comparability across video input settings (3Hso W3, EJFj W1/Q1b): Examining whether model comparisons and Gemini-3-Pro's lead hold under strictly identical frame-sampling and subtitle inputs.
  4. Evaluation reliability, human study details, and cross-family judge verification (3Hso W3/Q2, EJFj W2/Q2): Reporting sample counts and inter-annotator agreement for human verification and Appendix E, and testing an external judge outside the Qwen3.5 family to check ranking sensitivity and same-family judge preference.
  5. Practical relevance of deeply nested instructions and Figure 10 correction (3Hso W3/Q3, EJFj W2): Clarifying how deeply nested instructions relate to practical use cases and fixing the displayed example in Figure 10.

4. How We Addressed Reviewer Concerns and Questions

  • [G9hj W2/Q1, 3Hso W2/Q1, EJFj W1/Q1a] Separating Video Comprehension, Condition Judgment, Path Selection, and Execution: We conducted controlled interventions on conditional instructions comparing Isolated Condition Check, Original Conditional Prompt, With GT Video Description, With GT Condition Labels, and Target Branch Only on the same videos, target tasks, and checklists. While providing ground-truth video descriptions (+10.0 to +11.6 points in TISR) or exact condition truth labels (+19.0 to +23.7 points in TISR) recovers part of the errors, models still fail on 31.9%–52.5% of instructions even when condition truth labels are explicitly given, confirming that complex instruction following and conditional routing remain major bottlenecks. Furthermore, while Gemini-3-Pro and Gemma-4-31B-it perform almost identically on Target Branch Only (70.5% vs. 69.2% TISR), a large gap appears under With GT Condition Labels (68.1% vs. 47.5%), suggesting that handling multi-branch conditional structures plays a major role in the performance gap beyond simply recognizing Gemini-extracted condition facts (see Rebuttal to G9hj (1)–(2), 3Hso (2), EJFj (1), and Appendix A.4).

  • [G9hj W1/Q2, 3Hso W1] Novelty and Core Findings Beyond IF-VidCap: We clarified how Video-IFBench advances beyond IF-VidCap's short-clip (up to 1 minute) captioning via 700+ audio-visual videos (up to 10 minutes), 32 fine-grained tasks, and multi-stage conditional tree construction (Sec. 2.2, 3.2). Crucially, our controlled experiments reveal three core findings specific to video-grounded conditional execution: (1) even on the exact subset of cases where a model passes both Isolated Condition Check (100%) and Target Branch Only (100%) completely, it still fails on 26.9% (Gemini-3-Pro) and 57.0% (Gemma-4-31B-it) of full conditional prompts, with 71.8%–72.2% of those failures executing another branch's task instead; (2) conditional routing is bottlenecked primarily by rejecting near-miss false conditions (16.4% vs. 7.4% false-positive rate for Gemma-4-31B-it vs. Gemini-3-Pro) rather than recognizing true ones (84.5% vs. 85.0%), explaining the branch-count and position effects in Table 3; and (3) scaling model size alone is insufficient compared to explicit step-by-step verification of reasoning models (Tables 1 and 3b; see Rebuttal to G9hj (3), 3Hso (1), and Appendix A.4).

  • [3Hso W3, EJFj W1/Q1b] Controlled Comparison Under Identical Video and Subtitle Inputs: Beyond clarifying our long-video input setup (Appendix F), we re-evaluated 6 representative models across all Selection and Nested instructions under the exact same uniform 50-frame and subtitle inputs (without native audio), confirming that Gemini-3-Pro still leads Qwen3.5-397B-Think and other models by a wide margin (52.2% / 45.3% vs. 36.6% / 31.9% overall conditional TCSR / TISR) and relative model rankings remain consistent with Table 1 (see Rebuttal to 3Hso (3), EJFj (1), and Appendix A.5).

  • [3Hso W3/Q2, EJFj W1/W2/Q1a/Q2] Human Verification, Human Agreement Study, and Cross-Family Judge: We added full statistics on our two-stage human verification (discarding 54.2% of candidates with 81.9% inter-annotator agreement and refining ~25% of retained samples; Sec. 3.2) and human agreement study (280 model responses, 720 response evaluations, 2,878 constraint judgments, 88.4% inter-annotator agreement, and identical 5-model ranking under human and judge scores; Appendix E, Table 10). Moreover, re-evaluating over 9k paired responses across 12 representative models with GPT-5.4 as a cross-family judge yields 89.5% constraint agreement, 89.6% task-pass agreement, and Spearman rank correlations of 0.993 (TCSR) and 0.979 (TISR) with Qwen3.5-397B (see Rebuttal to 3Hso (3), EJFj (1)–(2), and Appendix A.3).

  • [3Hso W3/Q3, EJFj W2] Practical Relevance of Nested Instructions and Figure 10 Correction: We clarified the practical value of Nested instructions in rule-based workflows (such as video monitoring and anomaly detection) alongside deeper stress-testing (Sec. 3.1, Rebuttal to 3Hso (3)), and fixed the copy-paste mistake in Figure 10 (Appendix D, Rebuttal to EJFj (2)).

Once again, we sincerely thank AC and reviewers for your time and constructive feedback.

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.