acceptodds
Under review as a conference paper at ICLR 2027

IVU-Bench: Benchmarking Multimodal Large Language Models for Video Understanding under Incomplete Observations

Abstract

Multimodal large language models (MLLMs) have demonstrated strong video understanding, yet existing evaluations largely focus on recognizing explicitly observable content. Real-world video understanding often requires inference beyond observation, including predicting missing events, recovering spatial relations from partial evidence, and reasoning under modified or hypothetical conditions. We introduce IVU-Bench, a benchmark for evaluating MLLMs under incomplete observations, comprising 2,032 question-answer pairs, and 13 tasks across temporal, spatial, and factual dimensions. Temporal incompleteness involves inferring missing events or temporal structure; spatial incompleteness involves recovering object locations and relations from partial observations across time and viewpoints; and factual incompleteness involves reasoning about event dynamics under modified or hypothetical conditions absent from the original video. Questions from multiple task types are grounded in shared videos, enabling controlled cross-task comparisons while reducing content-related confounds. To assess whether correct answers are supported by appropriate visual evidence, IVU-Bench provides object-level spatiotemporal annotations for target evidence and semantically related or irrelevant distractors. We further evaluate models with three complementary metrics: Answer Accuracy, Evidence Discrimination, and Joint Answer–Evidence Score, measuring answer correctness, target-evidence preference, and answer–evidence consistency, respectively. Experiments across representative MLLMs reveal substantial weaknesses across all three dimensions, especially in spatial reasoning from partial visual evidence, highlighting the gap between recognizing visible content and inferring unobserved information.

Primary area:
datasets and benchmarks
Keywords:
Incomplete Observations, Video Understanding, Benchmarking

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_aK9mOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

IVU-Bench evaluates video understanding under incomplete observations using 2,032 eight-choice questions from 127 egocentric videos. Its 13 tasks span temporal, spatial, and factual reasoning. The authors also annotate candidate visual evidence and evaluate answer accuracy, evidence discrimination, and their joint score across human and MLLM baselines. Models perform especially poorly on spatial questions.

Critical strengths
  1. Shared-video design: Different tasks use the same videos, reducing content differences when comparing capabilities.
  2. Separate evidence evaluation: Candidate-level annotations reveal cases where a correct answer does not coincide with discriminative evidence selection.
Critical weaknesses
  1. Unclear construct validity: Some tasks, particularly ordering and object-position questions, may be solvable from directly visible information. The paper does not establish that each task actually requires inference under incomplete observation.
  2. Insufficient answer and baseline validation: Counterfactual “most likely” actions can have multiple plausible answers, including apparently near-equivalent options in the appendix. The paper does not report annotator agreement, text-only results, or enough human-baseline protocol detail to assess the reported gap.
Questions for authors
  1. Clarification: For each task, what answer-relevant information is unavailable to the model, and what fraction of questions can be answered from directly visible frames?
  2. Clarification: How were plausible alternative answers adjudicated in factual tasks? Please report independent annotator agreement and the text-only validation results.
  3. Clarification: Did human experts receive the same sampled frames and candidate evidence as models, and how many people answered each question?
4 comments

1.Clarification: For each task, what answer-relevant information is unavailable to the model, and what fraction of questions can be answered from directly visible frames?

Temporal tasks control different types of missing information. Prediction and Planning withhold the relevant future process, requiring the model to infer subsequent actions from observed states; Planning additionally provides a high-level goal. Abduction hides an intermediate process and requires reconstruction of the missing event from the preceding and following clips. Ordering shuffles the clips and requires recovery of their original temporal structure. Models receive the context needed for inference, while the event or ordering information corresponding to the answer is withheld by the task conditions.

Spatial tasks distribute object relations across local observations from different times and viewpoints. Models must first establish a reference frame using two designated objects, then locate the target object and infer its positions and relative relations before and after movement. The reference objects and the target object are not all fully visible in the same frame, so reading positions directly from one frame cannot substitute for cross-view integration. This is also why we evaluate object positions, position changes, and pairwise object relations on the same videos.

Factual tasks provide the process that actually occurred in the original video and modify facts or outcomes through text. Models must apply the video's objects, states, and process relations to the modified conditions to infer a hypothetical scenario absent from the original video. The visible video therefore establishes the factual basis, while the intervention in the question defines the new scenario to be inferred.

To examine the role of directly visible cues, we added single-frame and extended-observation experiments in Appendix F. Even with a manually annotated target-evidence frame, single-frame spatial accuracy remains 14.3%; by contrast, presenting clips in chronological order yields 95.3% accuracy on Ordering. The former is consistent with the need to integrate multiple views for spatial relations, while the latter confirms the importance of temporal-order cues for Ordering. These experiments validate tasks under controlled observation conditions. The fraction of questions answerable from directly visible cues is a question-level property and cannot simply be replaced by single-frame accuracy.

“Incompleteness” in IVU-Bench is defined by the information missing in each task: events or ordering in the temporal dimension, complete relations within a single observation in the spatial dimension, and the realized trajectory under modified conditions in the factual dimension. Models must recover this information from the partial observations provided. These definitions, task examples, and observation controls jointly establish that the benchmark evaluates inference of complete relations or unobserved states from partial evidence, rather than requiring questions to be answered independently of the video content.

2.Clarification: How were plausible alternative answers adjudicated in factual tasks? Please report independent annotator agreement and the text-only validation results.

Factual answers are judged by consistency among the video context, intervention conditions, and outcome. Candidate actions must be compatible with the video's objects and process states, the facts explicitly changed by the question, and the conditions that remain in force; OFM additionally requires consistency with the specified alternative outcome. Thus, an option that appears plausible out of context can be a distractor because it violates a specific condition. Twenty trained undergraduate and master's students annotated the data under unified guidelines, followed by cross-checking among annotators and subsequent verification by senior annotators to assess semantic and conditional consistency; detailed rules are provided in Appendix D. Text-only validation offers a complementary measure of the contribution of linguistic cues, with complete results in Appendix G, Table 10.

Model/combination abbreviations: Q = Qwen3-4B; P = Phi-4-mini-instruct; L = Llama-3.2-3B-Instruct; I = InternLM2.5-1.8B-Chat; M = MiniCPM-2B-sft.

Model/groupPred.Abd.Plan.Ord.Temp.Pos-BPos-AB-vs-APair-BPair-ASpa.SFPDFPCFIOFMFact.Avg.
Q51.1850.3949.6113.3941.1418.5021.6524.0222.8311.0219.6159.0659.0664.5753.5459.0639.93
P49.6145.6748.0315.7539.7624.0220.0814.9622.0514.1719.0657.4855.9165.3552.7657.8738.90
L43.3139.3744.0915.7535.6321.2619.2913.7816.5422.0518.5846.4655.1255.9148.0351.3835.20
I40.1628.3534.6510.2428.3518.5014.1715.7514.9616.5415.9839.3733.8647.2425.2036.4226.92
M43.3131.5033.0715.7530.9114.5716.1416.5415.7514.1715.4340.9450.3954.3315.7540.3528.90
Q+P+L20.4724.4126.772.3618.505.123.944.333.940.793.6229.9232.2842.5224.4132.2818.14
Q+P+I21.2613.3917.320.0012.991.971.970.791.570.001.2626.7718.9029.9211.0221.6511.97
Q+P+M21.2615.7516.540.0013.390.391.971.971.570.001.1825.9825.9836.223.1522.8312.47
Q+L+I15.7514.1716.540.7911.810.791.180.391.570.000.7922.8321.2627.5610.2420.4711.02
Q+L+M17.3216.5414.170.7912.200.391.971.181.570.791.1822.0525.9833.863.1521.2611.55
Q+I+M17.3210.2411.020.799.840.001.181.181.570.790.9419.6916.5423.622.3615.558.78
P+L+I17.3214.1717.321.5712.602.362.360.791.572.361.8924.4118.1129.9212.6021.2611.92
P+L+M18.1114.1717.322.3612.991.572.361.973.940.792.1322.8326.7735.432.3621.8512.32
P+I+M18.909.4513.392.3611.021.181.570.791.571.571.3419.6917.3227.563.1516.939.76
L+I+M15.758.6613.393.1510.241.572.362.361.570.791.7318.1116.5424.413.9415.759.24
Q+P+L+I13.3911.8115.750.0010.240.790.790.390.790.000.5521.2616.5425.988.6618.119.63
Q+P+L+M12.6013.3914.170.0010.040.390.791.181.570.000.7919.6920.4732.280.7918.319.71
Q+P+I+M14.967.8711.020.008.460.000.790.790.790.000.4717.3213.3922.050.7913.397.44
Q+L+I+M10.248.6610.240.007.280.000.790.390.790.000.3916.5414.1721.260.7913.196.96
P+L+I+M11.816.3011.811.577.870.791.180.391.570.790.9416.5413.3922.831.5713.587.47

2.Clarification: How were plausible alternative answers adjudicated in factual tasks? Please report independent annotator agreement and the text-only validation results.

Text-only results reveal differences within factual tasks. Across the five models, factual accuracy ranges from 36.42% to 59.06%, temporal accuracy from 28.35% to 41.14%, spatial accuracy from 15.43% to 19.61%, and CFI accuracy from 47.24% to 65.35%. The intersection of questions answered correctly by Qwen3, Phi, and Llama is 32.28% for the factual dimension, 42.52% for CFI, and 3.62% for the spatial dimension. OFM exhibits more pronounced differences across models, with Qwen3 at 53.54% and MiniCPM at 15.75%, for example. This shows that single-fact modifications, hypothetical injections, and alternative outcomes impose different reasoning demands. Organizing the benchmark into multiple tasks reveals differences in how models use linguistic commonsense and conditional constraints.

Answer adjudication and text-only validation address different questions: the former determines which option satisfies the video and intervention constraints, while the latter measures whether a model can predict the reference answer without watching the video. We therefore combine annotation verification, a text-only reference, and visual-evidence evaluation. For the near-synonymous options highlighted by the reviewer, the key criteria are whether the action completes the goal specified in the question and whether it satisfies the conditions unaffected by the intervention. These concrete criteria explain the distinction between reference answers and distractors more effectively than similarity of wording alone.

3.Clarification: Did human experts receive the same sampled frames and candidate evidence as models, and how many people answered each question?

In the original human evaluation, four participants each answered a subset of the questions, with one participant answering each question. They watched continuous video within the designated intervals and did not receive the evidence-candidate catalog used by models, so their sampled-frame and evidence inputs were not identical to those of the models. To maintain a consistent comparison in the main table, the current manuscript removes the human-expert row; the main conclusions are supported by 30 model configurations and the corresponding control experiments.

Reviewer_H2ogOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces IVU-Bench, a benchmark with 2032 questions from first-person videos covering temporal, spatial, and counterfactual reasoning tasks under incomplete observations. It evaluates both answer accuracy and the selection of supporting visual evidence from provided candidates. Experiments show that current models perform substantially below the reported human baseline, especially on spatial tasks.

Critical strengths
  1. The benchmark evaluates supporting evidence, not just final answers. Its annotations distinguish useful evidence from related and unrelated distractors, helping reveal errors that answer accuracy alone may miss.

  2. It brings temporal, spatial, and counterfactual tasks together on shared real-world videos. This provides a common setting for comparing different abilities while reducing differences in video content.

Critical weaknesses
  1. Some tasks are not clearly distinguished. In the appendix’s Prediction and Planning example, the question, displayed frames, and answer choices are identical, with no explicit planning goal. This raises doubts about whether these tasks consistently measure different abilities.

  2. The need for video-based reasoning is insufficiently demonstrated. Text-only validation is mentioned, but its results are not reported. Single-frame and matched complete-observation controls are also missing, making it difficult to separate reasoning about missing information from language shortcuts or basic perception errors.

  3. Annotation quality and the human comparison are insufficiently documented. The paper does not report annotation agreement or clearly describe the human evaluation protocol, including whether humans received the same restricted observations as models. This weakens confidence in both the answer labels and the reported human–model gap.

  4. Most of the evaluated open-source models is Qwen falimy. More evalutaions on diverse models are requried for better validation.

  5. Some questions have ambiguous answers. The phone-call example contains multiple similarly reasonable responses, but only one is marked correct. Without clear rules for resolving such cases, the benchmark may penalize valid answers rather than reasoning errors.

Questions for authors

Please refer to Weaknesses

6 comments
  1. Some tasks are not clearly distinguished. In the appendix’s Prediction and Planning example, the question, displayed frames, and answer choices are identical, with no explicit planning goal. This raises doubts about whether these tasks consistently measure different abilities.

Prediction and Planning differ in whether a high-level task goal is provided. In addition to the observations, Planning supplies a task field and asks the model to select the subsequent action that achieves that goal; Prediction asks for the most likely action given the current observations. We have added the task field to the Planning example in the appendix to make this goal conditioning explicit.

  1. The need for video-based reasoning is insufficiently demonstrated. Text-only validation is mentioned, but its results are not reported. Single-frame and matched complete-observation controls are also missing, making it difficult to separate reasoning about missing information from language shortcuts or basic perception errors.

We analyze the roles of linguistic cues, local visual cues, and temporal information through three complementary conditions: text-only, single-frame, and extended observation.

Model/combination abbreviations: Q = Qwen3-4B; P = Phi-4-mini-instruct; L = Llama-3.2-3B-Instruct; I = InternLM2.5-1.8B-Chat; M = MiniCPM-2B-sft.

Model/groupPred.Abd.Plan.Ord.Temp.Pos-BPos-AB-vs-APair-BPair-ASpa.SFPDFPCFIOFMFact.Avg.
Q51.1850.3949.6113.3941.1418.5021.6524.0222.8311.0219.6159.0659.0664.5753.5459.0639.93
P49.6145.6748.0315.7539.7624.0220.0814.9622.0514.1719.0657.4855.9165.3552.7657.8738.90
L43.3139.3744.0915.7535.6321.2619.2913.7816.5422.0518.5846.4655.1255.9148.0351.3835.20
I40.1628.3534.6510.2428.3518.5014.1715.7514.9616.5415.9839.3733.8647.2425.2036.4226.92
M43.3131.5033.0715.7530.9114.5716.1416.5415.7514.1715.4340.9450.3954.3315.7540.3528.90
Q+P+L20.4724.4126.772.3618.505.123.944.333.940.793.6229.9232.2842.5224.4132.2818.14
Q+P+I21.2613.3917.320.0012.991.971.970.791.570.001.2626.7718.9029.9211.0221.6511.97
Q+P+M21.2615.7516.540.0013.390.391.971.971.570.001.1825.9825.9836.223.1522.8312.47
Q+L+I15.7514.1716.540.7911.810.791.180.391.570.000.7922.8321.2627.5610.2420.4711.02
Q+L+M17.3216.5414.170.7912.200.391.971.181.570.791.1822.0525.9833.863.1521.2611.55
Q+I+M17.3210.2411.020.799.840.001.181.181.570.790.9419.6916.5423.622.3615.558.78
P+L+I17.3214.1717.321.5712.602.362.360.791.572.361.8924.4118.1129.9212.6021.2611.92
P+L+M18.1114.1717.322.3612.991.572.361.973.940.792.1322.8326.7735.432.3621.8512.32
P+I+M18.909.4513.392.3611.021.181.570.791.571.571.3419.6917.3227.563.1516.939.76
L+I+M15.758.6613.393.1510.241.572.362.361.570.791.7318.1116.5424.413.9415.759.24
Q+P+L+I13.3911.8115.750.0010.240.790.790.390.790.000.5521.2616.5425.988.6618.119.63
Q+P+L+M12.6013.3914.170.0010.040.390.791.181.570.000.7919.6920.4732.280.7918.319.71
Q+P+I+M14.967.8711.020.008.460.000.790.790.790.000.4717.3213.3922.050.7913.397.44
Q+L+I+M10.248.6610.240.007.280.000.790.390.790.000.3916.5414.1721.260.7913.196.96
P+L+I+M11.816.3011.811.577.870.791.180.391.570.790.9416.5413.3922.831.5713.587.47

2.The need for video-based reasoning is insufficiently demonstrated. Text-only validation is mentioned, but its results are not reported. Single-frame and matched complete-observation controls are also missing, making it difficult to separate reasoning about missing information from language shortcuts or basic perception errors.

Appendix G, Table 10 reports text-only results for five models on all retained questions: Avg. ranges from 26.92% to 39.93%, and Overall from 24.21% to 35.19%. Differences across dimensions are clear: factual accuracy is 36.42%–59.06%, temporal accuracy 28.35%–41.14%, and spatial accuracy 15.43%–19.61%; CFI reaches 47.24%–65.35%, while Ordering yields 10.24%–15.75%. These results quantify the support that linguistic commonsense provides for each task and establish a reference for evaluation with visual input. Model combinations use the intersection of correctly answered samples: Qwen3, Phi, and Llama yield an Overall of 14.67% and an Avg. of 18.14%, with intersection rates of 32.28%, 3.62%, and 2.36% for the factual dimension, spatial dimension, and Ordering. This combination analysis further indicates that shared textual cues are concentrated in factual tasks, whereas specific spatial relations and temporal order exhibit different patterns. In the single-frame condition in Appendix F, spatial accuracy is 14.3%, compared with 25.6% under the multi-frame EC-3 condition; restoring clip order raises Ordering accuracy to 95.3%. These changes across observation conditions are consistent with the spatial integration and temporal-structure reconstruction targeted by the benchmark. The evidence interfaces and task coverage for each control are specified in Appendix F.

Reviewer_UhMnOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces IVU-Bench, a benchmark for evaluating multimodal large language models on video understanding under incomplete observation. The authors formalize the setting as IVU and split it into three dimensions. Temporal incompleteness covers predicting the unobserved future, reconstructing an intermediate process that has been cut out, goal-conditioned planning, and recovering a shuffled order. Spatial incompleteness asks the model to recover object positions and pairwise relations from partial observations spread across time and viewpoints. Factual incompleteness asks for reasoning under hypothetical conditions: Single-/Dual-Fact Perturbation, Counterfactual Fact Injection, and Outcome Fact Modification. The benchmark contains 127 egocentric videos (EPIC-KITCHENS-100, EPIC-KITCHENS-55, HD-EPIC, Ego4D) and 2,032 eight-way multiple-choice QA pairs. Each question has a [ts, te] interval that limits the visible context, and each video carries 16 questions spanning the dimensions to control for content confounds. Every question except those in Ordering comes with object-level spatiotemporal evidence annotations in three categories: target evidence, semantically related distractors, and semantically irrelevant distractors. The authors propose three metrics. Answer Accuracy is the rate of correct answers. Evidence Discrimination D is the target-evidence selection rate minus the distractor selection rate. Joint Answer–Evidence Score J is the fraction of questions answered correctly with positive evidence discrimination; the paper explicitly defines it as answer–evidence consistency rather than causal faithfulness of the reasoning. The experiments cover commercial and open-source models and include an inter-task correlation analysis, plus comparisons across model scale, frame budget, and Thinking vs. Instruct variants.

Critical strengths
  1. The problem setting is realistic, and the three dimensions are substantively distinct. The reason information is missing really does differ across them: a gap in time, incomplete coverage caused by how the viewpoints are distributed, and a hypothetical intervention introduced by the question. The paper does more than relabel existing tasks. It takes their common requirement, inferring states that were never observed, and organizes it into a structure that can be evaluated. Each task within a dimension also targets a specific form of reasoning (abduction and prediction address temporal gaps in different directions).

  2. Decoupling answers from evidence is the most valuable contribution of the paper. The joint evaluation exposes behavior that accuracy alone cannot show. Qwen3.7-Plus has higher answer accuracy than GPT-5.5 (54.7% vs 52.1%) but lower D and J (48.4/32.7 vs 56.0/35.8). On the spatial dimension, GPT-5.5 reaches a D of 57.8 but a J of only 20.1, which means the model can identify the target observations to some degree but cannot integrate them into a correct spatial inference. This dissociation, where the model picks the right evidence and still reasons to the wrong answer, is a substantive advance over existing grounded VideoQA work.

  3. The shared-video design is a serious control for confounds. Each video carries 16 questions across the temporal, spatial, and factual dimensions, so cross-task comparisons rest on the same visual content. This is uncommon in benchmark papers, and the paper explicitly uses it to support the validity of its cross-task analysis.

  4. The human baseline and the choice of analyses show restraint. Human experts reach 88.2% overall, and the paper reports this per dimension (Temporal 89.7 / Spatial 87.8 / Factual 87.0), which shows that the tasks are solvable instead of offering a single aggregate number. The frame-budget experiment (going from 64→128 frames changes results by only −0.24 to +0.90) is kept as a negative result and used to argue that sampling density is not the bottleneck. The Thinking vs. Instruct comparison states explicitly that it "does not establish causal faithfulness or isolate the effect of additional inference-time computation". The partial correlation analysis in §5.1 likewise says up front that its estimates "do not establish statistical significance".

  5. The appendix gives a full breakdown of the evidence metrics. Target-evidence selection rates, related and irrelevant distractor selection rates, and the resulting discrimination difference are all reported per task, so the diagnosis that models tend to keep distractors can be checked item by item. For example, GPT-5.5 selects 82.1% of target evidence on average, while also selecting 30.5% of related distractors and 21.6% of irrelevant distractors.

Critical weaknesses

W1 (decisive) The benchmark's validation is claimed, but no results are reported

The authors write in §3.3:

"We apply two complementary quality-control procedures. Text-only validation identifies questions that can be solved without visual input due to language priors or option artifacts. Inter-annotator verification cross-checks questions, answers, and annotations against the source videos."

However, neither procedure comes with a single number in the main text or the appendix:

  • Text-only validation: The paper does not say how many questions this step identified and removed (or kept), and it does not report text-only accuracy on the remaining questions.
  • Inter-annotator verification: The number of annotators, their qualifications, agreement rates on questions, answers, and evidence, κ values, and how disagreements were resolved are all left unstated.
  • Answer uniqueness in the factual dimension: Counterfactual Fact Injection and Outcome Fact Modification can in principle have more than one self-consistent continuation, yet eight-way multiple choice forces a single answer. The paper does not explain how such ambiguity was adjudicated and reports no per-dimension agreement rates.
  • The shortcut upper bound is unknown. Chance level for eight-way multiple choice is 12.5%, but the paper gives no shortcut baseline (text-only, single-frame, or answering from the option construction alone), so none of the accuracies can be compared with what a model scores without watching the video.

For a benchmark paper, the validity of the benchmark is the entire conclusion, so what is missing here is support for the conclusion itself, not an experimental detail.

Requested from the authors: (1) the concrete results of text-only validation and the number of questions handled on that basis; (2) the number of annotators, the agreement rates (including κ, reported per dimension) for questions, answers, and evidence, and the procedure for adjudicating ambiguity in the factual dimension; (3) results for at least one shortcut baseline.

W2 (important) The cross-dimension gap is a main finding, but it is not analyzed and task-format confounds are not ruled out

The abstract, introduction, and conclusion present the claim that spatial reasoning is particularly challenging as one of the paper's core findings, but the paper contains no dedicated cross-dimension analysis:

  • No cross-model consistency check. The paper does not report whether this difficulty ordering holds across the 24 configurations.
  • The three dimensions are not comparable in task format. The [ts, te] interval of a spatial question usually covers only a local segment, while factual questions often need the whole video. Spatial questions ask for coordinate-style relation judgments, while factual questions involve text-based reasoning about interventions. The spatial dimension has 5 tasks against 4 each for temporal and factual, so the macro-average weights differ too. The result that spatial is worst may come partly from task format and not from incomplete observation itself.
  • The most direct evidence goes unused. The human baseline barely changes across the three dimensions (87.0–89.7, a spread of 2.7 percentage points), while the models span 43.4 percentage points (31.8–75.2). This shows that the gap is a failure mode specific to models rather than intrinsic task difficulty. §5.2 also reports only on the aggregate metrics that going from 64→128 frames changes almost nothing (−0.24 to +0.90). If spatial tasks are hard because the evidence is spread over several viewpoints, the per-dimension frame-budget effect would test that explanation directly, but it is not reported.

Requested from the authors: whether the difficulty ordering holds across the 24 configurations, an analysis of the task-format confounds (or a discussion of them in the Limitations), the human-versus-model comparison by dimension, and the frame-budget effect broken down by dimension.

W3 (important) The evidence metrics admit a degenerate solution, and the statistics ignore that questions are not independent

  • Evidence selection is unlimited in number. The model returns the IDs of the evidence candidates it chooses, "without being constrained to select a fixed number of candidates". D and J can therefore be pushed up by a broad select-all strategy, which pays off whenever the gain in target-evidence selection rate outweighs the cost in distractor selection rate. The paper reports neither the D and J of selecting every candidate nor a control protocol with a fixed selection size (e.g., top-1 / top-3).
  • No error bars or confidence intervals. Every value in the main tables is a single point estimate. Without a variance reference, a frame-budget change within 0.9 percentage points cannot be read as "no effect".
  • Neither questions nor configurations are independent. The 16 questions from the same video share visual content and narrative structure, so the 2,032 samples are not independent observations, yet none of the reported accuracies come with confidence intervals or standard errors clustered by video. The correlations in §5.1 are computed over 24 configurations that include 64/128-frame pairs of the same model. The authors note that "configurations from the same model families are related", but they make no adjustment for it.

Requested from the authors: D/J scores for the select-all strategy or a control setting with a fixed selection size, and confidence intervals clustered by video.

Questions for authors
  1. How many questions did text-only validation identify as answerable from language priors or option construction? Were they removed or kept? What is the text-only accuracy on the remaining questions?
  2. How many annotators were there? What are the agreement rates and κ for question correctness, answers, and evidence annotations? How were disagreements resolved?
  3. If a model selects every evidence candidate, what are its D and J? Do you have results under a protocol with a fixed selection size?
  4. Why do the human baseline and the models show such different difficulty patterns across the three dimensions (a 2.7-point spread for humans, 43.4 points for models)? Does this mean the gap comes from a specific capability gap in the models and not from the intrinsic difficulty of the tasks?
  5. What is the frame-budget effect (64 vs 128) for each dimension separately? Does the spatial dimension follow a different trend from the other two?
  6. Given that questions are not independent of their videos, what are the confidence intervals when clustered by video? Do the dimension gaps in the main table still hold once the clustered variance is taken into account?
9 comments
  1. How many questions did text-only validation identify as answerable from language priors or option construction? Were they removed or kept? What is the text-only accuracy on the remaining questions?

We have added text-only validation results for five models on the final question set in Appendix G, Table 10. Dataset construction began with more than 3,000 candidate questions. Combining text-bias checks with manual quality checks, we retained 127 videos and 2,032 questions. Each video has 16 questions spanning 13 task types, with each of the three object-position tasks instantiated for two objects. To maintain complete task coverage within each video, screening was performed at the level of videos and their associated questions, removing more than 1,000 questions in total. For the final retained set, models received only the original questions and options A–H, without videos, images, or evidence candidates. Across the five models, Avg., which equally weights the dimensions, ranges from 26.92% to 39.93%, while Overall, which weights individual questions equally, ranges from 24.21% to 35.19%. The former matches the equal weighting of the three dimensions in the main table; the latter reflects the proportion of correctly answered questions across the full set.

Model/combination abbreviations: Q = Qwen3-4B; P = Phi-4-mini-instruct; L = Llama-3.2-3B-Instruct; I = InternLM2.5-1.8B-Chat; M = MiniCPM-2B-sft.

Model/groupPred.Abd.Plan.Ord.Temp.Pos-BPos-AB-vs-APair-BPair-ASpa.SFPDFPCFIOFMFact.Avg.
Q51.1850.3949.6113.3941.1418.5021.6524.0222.8311.0219.6159.0659.0664.5753.5459.0639.93
P49.6145.6748.0315.7539.7624.0220.0814.9622.0514.1719.0657.4855.9165.3552.7657.8738.90
L43.3139.3744.0915.7535.6321.2619.2913.7816.5422.0518.5846.4655.1255.9148.0351.3835.20
I40.1628.3534.6510.2428.3518.5014.1715.7514.9616.5415.9839.3733.8647.2425.2036.4226.92
M43.3131.5033.0715.7530.9114.5716.1416.5415.7514.1715.4340.9450.3954.3315.7540.3528.90
Q+P+L20.4724.4126.772.3618.505.123.944.333.940.793.6229.9232.2842.5224.4132.2818.14
Q+P+I21.2613.3917.320.0012.991.971.970.791.570.001.2626.7718.9029.9211.0221.6511.97
Q+P+M21.2615.7516.540.0013.390.391.971.971.570.001.1825.9825.9836.223.1522.8312.47
Q+L+I15.7514.1716.540.7911.810.791.180.391.570.000.7922.8321.2627.5610.2420.4711.02
Q+L+M17.3216.5414.170.7912.200.391.971.181.570.791.1822.0525.9833.863.1521.2611.55
Q+I+M17.3210.2411.020.799.840.001.181.181.570.790.9419.6916.5423.622.3615.558.78
P+L+I17.3214.1717.321.5712.602.362.360.791.572.361.8924.4118.1129.9212.6021.2611.92
P+L+M18.1114.1717.322.3612.991.572.361.973.940.792.1322.8326.7735.432.3621.8512.32
P+I+M18.909.4513.392.3611.021.181.570.791.571.571.3419.6917.3227.563.1516.939.76
L+I+M15.758.6613.393.1510.241.572.362.361.570.791.7318.1116.5424.413.9415.759.24
Q+P+L+I13.3911.8115.750.0010.240.790.790.390.790.000.5521.2616.5425.988.6618.119.63
Q+P+L+M12.6013.3914.170.0010.040.390.791.181.570.000.7919.6920.4732.280.7918.319.71
Q+P+I+M14.967.8711.020.008.460.000.790.790.790.000.4717.3213.3922.050.7913.397.44
Q+L+I+M10.248.6610.240.007.280.000.790.390.790.000.3916.5414.1721.260.7913.196.96
P+L+I+M11.816.3011.811.577.870.791.180.391.570.790.9416.5413.3922.831.5713.587.47
  1. How many questions did text-only validation identify as answerable from language priors or option construction? Were they removed or kept? What is the text-only accuracy on the remaining questions?

The text-only results reveal clear differences across tasks: accuracy ranges from 28.35% to 41.14% in the temporal dimension, 15.43% to 19.61% in the spatial dimension, and 36.42% to 59.06% in the factual dimension. Counterfactual Fact Injection reaches 47.24%–65.35%, whereas Ordering yields 10.24%–15.75%. Factual interventions and action sequences contain commonsense and semantic relations that can support inference, whereas scene-specific spatial coordinates and shuffled clip IDs depend more on scene information. These results provide an explicit nonvisual reference for each task, helping readers distinguish performance supported by language priors from performance that requires further assessment using visual evidence.

Model combinations further characterize the language priors shared across models. A combination reports the intersection of samples answered correctly by all its members: Qwen3, Phi, and Llama achieve an Overall of 14.67% and an Avg. of 18.14%, with intersection rates of 32.28%, 3.62%, and 2.36% for the factual dimension, spatial dimension, and Ordering, respectively. Avg. ranges from 8.78% to 18.14% for three-model combinations and from 6.96% to 9.71% for four-model combinations. These differences indicate that shared linguistic cues are concentrated primarily in factual tasks, while specific spatial relations and clip order exhibit different answerability patterns. Individual-model results quantify text-only performance, whereas combination results identify jointly answerable samples. This also illustrates the value of reporting both answer accuracy and evidence metrics in IVU-Bench: evaluating not only whether an answer is correct, but also whether it is accompanied by effective discrimination of supporting evidence.

  1. How many annotators were there? What are the agreement rates and κ for question correctness, answers, and evidence annotations? How were disagreements resolved?

Annotation was performed by 20 trained undergraduate and master's students. We used standardized training and written annotation guidelines to minimize disagreements and maintain consistency. For temporal tasks, we specified how actions should be extracted and organized them into three levels: high-level actions represent the final goal, such as making a cake; mid-level actions represent intermediate goals, such as preparing eggs, milk, or cooking equipment; and low-level actions are concrete operations, such as picking up or placing an object. For spatial tasks, the two objects anchoring the coordinate system and the target object must not all appear in the same frame. We also strictly specify the structure of the options: the first component denotes horizontal direction and the second denotes vertical height. For factual tasks, annotators must ground their annotations in the actual processes shown in the video, applying perturbations, injections, and outcome modifications to the observed action sequence. Cross-checking among annotators promotes a shared interpretation and minimizes disagreements, followed by verification by more senior students to further ensure consistency. Detailed annotation constraints are provided in Appendix D.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.