- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
IVU-Bench evaluates video understanding under incomplete observations using 2,032 eight-choice questions from 127 egocentric videos. Its 13 tasks span temporal, spatial, and factual reasoning. The authors also annotate candidate visual evidence and evaluate answer accuracy, evidence discrimination, and their joint score across human and MLLM baselines. Models perform especially poorly on spatial questions.
Critical strengths
- Shared-video design: Different tasks use the same videos, reducing content differences when comparing capabilities.
- Separate evidence evaluation: Candidate-level annotations reveal cases where a correct answer does not coincide with discriminative evidence selection.
Critical weaknesses
- Unclear construct validity: Some tasks, particularly ordering and object-position questions, may be solvable from directly visible information. The paper does not establish that each task actually requires inference under incomplete observation.
- Insufficient answer and baseline validation: Counterfactual “most likely” actions can have multiple plausible answers, including apparently near-equivalent options in the appendix. The paper does not report annotator agreement, text-only results, or enough human-baseline protocol detail to assess the reported gap.
Questions for authors
- Clarification: For each task, what answer-relevant information is unavailable to the model, and what fraction of questions can be answered from directly visible frames?
- Clarification: How were plausible alternative answers adjudicated in factual tasks? Please report independent annotator agreement and the text-only validation results.
- Clarification: Did human experts receive the same sampled frames and candidate evidence as models, and how many people answered each question?
4 comments
Q2 Part1
2.Clarification: How were plausible alternative answers adjudicated in factual tasks? Please report independent annotator agreement and the text-only validation results.
Factual answers are judged by consistency among the video context, intervention conditions, and outcome. Candidate actions must be compatible with the video's objects and process states, the facts explicitly changed by the question, and the conditions that remain in force; OFM additionally requires consistency with the specified alternative outcome. Thus, an option that appears plausible out of context can be a distractor because it violates a specific condition. Twenty trained undergraduate and master's students annotated the data under unified guidelines, followed by cross-checking among annotators and subsequent verification by senior annotators to assess semantic and conditional consistency; detailed rules are provided in Appendix D. Text-only validation offers a complementary measure of the contribution of linguistic cues, with complete results in Appendix G, Table 10.
Model/combination abbreviations: Q = Qwen3-4B; P = Phi-4-mini-instruct; L = Llama-3.2-3B-Instruct; I = InternLM2.5-1.8B-Chat; M = MiniCPM-2B-sft.
| Model/group | Pred. | Abd. | Plan. | Ord. | Temp. | Pos-B | Pos-A | B-vs-A | Pair-B | Pair-A | Spa. | SFP | DFP | CFI | OFM | Fact. | Avg. |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Q | 51.18 | 50.39 | 49.61 | 13.39 | 41.14 | 18.50 | 21.65 | 24.02 | 22.83 | 11.02 | 19.61 | 59.06 | 59.06 | 64.57 | 53.54 | 59.06 | 39.93 |
| P | 49.61 | 45.67 | 48.03 | 15.75 | 39.76 | 24.02 | 20.08 | 14.96 | 22.05 | 14.17 | 19.06 | 57.48 | 55.91 | 65.35 | 52.76 | 57.87 | 38.90 |
| L | 43.31 | 39.37 | 44.09 | 15.75 | 35.63 | 21.26 | 19.29 | 13.78 | 16.54 | 22.05 | 18.58 | 46.46 | 55.12 | 55.91 | 48.03 | 51.38 | 35.20 |
| I | 40.16 | 28.35 | 34.65 | 10.24 | 28.35 | 18.50 | 14.17 | 15.75 | 14.96 | 16.54 | 15.98 | 39.37 | 33.86 | 47.24 | 25.20 | 36.42 | 26.92 |
| M | 43.31 | 31.50 | 33.07 | 15.75 | 30.91 | 14.57 | 16.14 | 16.54 | 15.75 | 14.17 | 15.43 | 40.94 | 50.39 | 54.33 | 15.75 | 40.35 | 28.90 |
| Q+P+L | 20.47 | 24.41 | 26.77 | 2.36 | 18.50 | 5.12 | 3.94 | 4.33 | 3.94 | 0.79 | 3.62 | 29.92 | 32.28 | 42.52 | 24.41 | 32.28 | 18.14 |
| Q+P+I | 21.26 | 13.39 | 17.32 | 0.00 | 12.99 | 1.97 | 1.97 | 0.79 | 1.57 | 0.00 | 1.26 | 26.77 | 18.90 | 29.92 | 11.02 | 21.65 | 11.97 |
| Q+P+M | 21.26 | 15.75 | 16.54 | 0.00 | 13.39 | 0.39 | 1.97 | 1.97 | 1.57 | 0.00 | 1.18 | 25.98 | 25.98 | 36.22 | 3.15 | 22.83 | 12.47 |
| Q+L+I | 15.75 | 14.17 | 16.54 | 0.79 | 11.81 | 0.79 | 1.18 | 0.39 | 1.57 | 0.00 | 0.79 | 22.83 | 21.26 | 27.56 | 10.24 | 20.47 | 11.02 |
| Q+L+M | 17.32 | 16.54 | 14.17 | 0.79 | 12.20 | 0.39 | 1.97 | 1.18 | 1.57 | 0.79 | 1.18 | 22.05 | 25.98 | 33.86 | 3.15 | 21.26 | 11.55 |
| Q+I+M | 17.32 | 10.24 | 11.02 | 0.79 | 9.84 | 0.00 | 1.18 | 1.18 | 1.57 | 0.79 | 0.94 | 19.69 | 16.54 | 23.62 | 2.36 | 15.55 | 8.78 |
| P+L+I | 17.32 | 14.17 | 17.32 | 1.57 | 12.60 | 2.36 | 2.36 | 0.79 | 1.57 | 2.36 | 1.89 | 24.41 | 18.11 | 29.92 | 12.60 | 21.26 | 11.92 |
| P+L+M | 18.11 | 14.17 | 17.32 | 2.36 | 12.99 | 1.57 | 2.36 | 1.97 | 3.94 | 0.79 | 2.13 | 22.83 | 26.77 | 35.43 | 2.36 | 21.85 | 12.32 |
| P+I+M | 18.90 | 9.45 | 13.39 | 2.36 | 11.02 | 1.18 | 1.57 | 0.79 | 1.57 | 1.57 | 1.34 | 19.69 | 17.32 | 27.56 | 3.15 | 16.93 | 9.76 |
| L+I+M | 15.75 | 8.66 | 13.39 | 3.15 | 10.24 | 1.57 | 2.36 | 2.36 | 1.57 | 0.79 | 1.73 | 18.11 | 16.54 | 24.41 | 3.94 | 15.75 | 9.24 |
| Q+P+L+I | 13.39 | 11.81 | 15.75 | 0.00 | 10.24 | 0.79 | 0.79 | 0.39 | 0.79 | 0.00 | 0.55 | 21.26 | 16.54 | 25.98 | 8.66 | 18.11 | 9.63 |
| Q+P+L+M | 12.60 | 13.39 | 14.17 | 0.00 | 10.04 | 0.39 | 0.79 | 1.18 | 1.57 | 0.00 | 0.79 | 19.69 | 20.47 | 32.28 | 0.79 | 18.31 | 9.71 |
| Q+P+I+M | 14.96 | 7.87 | 11.02 | 0.00 | 8.46 | 0.00 | 0.79 | 0.79 | 0.79 | 0.00 | 0.47 | 17.32 | 13.39 | 22.05 | 0.79 | 13.39 | 7.44 |
| Q+L+I+M | 10.24 | 8.66 | 10.24 | 0.00 | 7.28 | 0.00 | 0.79 | 0.39 | 0.79 | 0.00 | 0.39 | 16.54 | 14.17 | 21.26 | 0.79 | 13.19 | 6.96 |
| P+L+I+M | 11.81 | 6.30 | 11.81 | 1.57 | 7.87 | 0.79 | 1.18 | 0.39 | 1.57 | 0.79 | 0.94 | 16.54 | 13.39 | 22.83 | 1.57 | 13.58 | 7.47 |
Q2 Part2
2.Clarification: How were plausible alternative answers adjudicated in factual tasks? Please report independent annotator agreement and the text-only validation results.
Text-only results reveal differences within factual tasks. Across the five models, factual accuracy ranges from 36.42% to 59.06%, temporal accuracy from 28.35% to 41.14%, spatial accuracy from 15.43% to 19.61%, and CFI accuracy from 47.24% to 65.35%. The intersection of questions answered correctly by Qwen3, Phi, and Llama is 32.28% for the factual dimension, 42.52% for CFI, and 3.62% for the spatial dimension. OFM exhibits more pronounced differences across models, with Qwen3 at 53.54% and MiniCPM at 15.75%, for example. This shows that single-fact modifications, hypothetical injections, and alternative outcomes impose different reasoning demands. Organizing the benchmark into multiple tasks reveals differences in how models use linguistic commonsense and conditional constraints.
Answer adjudication and text-only validation address different questions: the former determines which option satisfies the video and intervention constraints, while the latter measures whether a model can predict the reference answer without watching the video. We therefore combine annotation verification, a text-only reference, and visual-evidence evaluation. For the near-synonymous options highlighted by the reviewer, the key criteria are whether the action completes the goal specified in the question and whether it satisfies the conditions unaffected by the intervention. These concrete criteria explain the distinction between reference answers and distractors more effectively than similarity of wording alone.
Q3
3.Clarification: Did human experts receive the same sampled frames and candidate evidence as models, and how many people answered each question?
In the original human evaluation, four participants each answered a subset of the questions, with one participant answering each question. They watched continuous video within the designated intervals and did not receive the evidence-candidate catalog used by models, so their sampled-frame and evidence inputs were not identical to those of the models. To maintain a consistent comparison in the main table, the current manuscript removes the human-expert row; the main conclusions are supported by 30 model configurations and the corresponding control experiments.
Q1
1.Clarification: For each task, what answer-relevant information is unavailable to the model, and what fraction of questions can be answered from directly visible frames?
Temporal tasks control different types of missing information. Prediction and Planning withhold the relevant future process, requiring the model to infer subsequent actions from observed states; Planning additionally provides a high-level goal. Abduction hides an intermediate process and requires reconstruction of the missing event from the preceding and following clips. Ordering shuffles the clips and requires recovery of their original temporal structure. Models receive the context needed for inference, while the event or ordering information corresponding to the answer is withheld by the task conditions.
Spatial tasks distribute object relations across local observations from different times and viewpoints. Models must first establish a reference frame using two designated objects, then locate the target object and infer its positions and relative relations before and after movement. The reference objects and the target object are not all fully visible in the same frame, so reading positions directly from one frame cannot substitute for cross-view integration. This is also why we evaluate object positions, position changes, and pairwise object relations on the same videos.
Factual tasks provide the process that actually occurred in the original video and modify facts or outcomes through text. Models must apply the video's objects, states, and process relations to the modified conditions to infer a hypothetical scenario absent from the original video. The visible video therefore establishes the factual basis, while the intervention in the question defines the new scenario to be inferred.
To examine the role of directly visible cues, we added single-frame and extended-observation experiments in Appendix F. Even with a manually annotated target-evidence frame, single-frame spatial accuracy remains 14.3%; by contrast, presenting clips in chronological order yields 95.3% accuracy on Ordering. The former is consistent with the need to integrate multiple views for spatial relations, while the latter confirms the importance of temporal-order cues for Ordering. These experiments validate tasks under controlled observation conditions. The fraction of questions answerable from directly visible cues is a question-level property and cannot simply be replaced by single-frame accuracy.
“Incompleteness” in IVU-Bench is defined by the information missing in each task: events or ordering in the temporal dimension, complete relations within a single observation in the spatial dimension, and the realized trajectory under modified conditions in the factual dimension. Models must recover this information from the partial observations provided. These definitions, task examples, and observation controls jointly establish that the benchmark evaluates inference of complete relations or unobserved states from partial evidence, rather than requiring questions to be answered independently of the video content.