acceptodds
Under review as a conference paper at ICLR 2027

Frames Don't Add Up: A Factorial Causal Audit of Contextual Evidence Utility in Video-Language Models

Abstract

Video question answering systems often select frames by scoring each independently for relevance, change, or diversity. Yet the answerer sees a set, not a ranking. Does a frame retain its value beside the other selected frames? We test this assumption through controlled downstream interventions. Our "Factorial Evidence Audit" holds the frame budget, surrounding frames, candidate slots, pixels, prompt, and answer scoring fixed while measuring individual and joint effects across roughly one million calls, six open video-language models, and three evidence-grounded benchmarks. Annotation-defined frames are causally useful, exceeding matched outside-evidence frames in 17 of 18 model–benchmark settings, with clustered intervals excluding zero in 16. But their values do not add independently. Evidence pairs interact more strongly than matched outside-evidence pairs in every setting (median absolute-interaction gap gold-vs-rest log odds), even after adjusting for the scale of their individual effects. This gap remains positive under chronological re-sorting and controls for candidate order and slot, score scale, context, frame budget, distractor construction, and RGB decoding. Deterministic scoring makes zero the exact additive prediction, and no-op controls return exactly zero. On LVBench and HERBench, common pointwise proxies poorly track measured utility; third- and fourth-order terms carry 23.5–27.7% of the absolute nonconstant M\"obius mass. A four-benchmark selector census exposes the practical consequence: in none of its 18 grounded settings does the method retrieving the most annotated evidence help the answerer most. Evidence matters. But fixed-budget video evidence selection is a set problem, not a ranking problem.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
video question answering, video-language models, frame selection, evidence selection, contextual utility, factorial causal audit, non-additivity, interaction effects, long-video understanding

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_VGWLOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies whether the benefits of individual frames add up in video question answering. It also measures the loss from selecting frames based on their individual scores. The authors use controlled experiments with fixed frame budgets, surrounding frames, and candidate positions. They measure individual frame benefits and interactions through changes in the model’s score for the correct answer.

Critical strengths
  1. The core experiments cover six models and three datasets with annotated evidence. The results show that evidence frames are more useful, but their combined benefits often differ from the sum of their individual benefits.

  2. The paper tests selecting three frames from four candidates. Using individual frame scores alone finds the highest-scoring combination in 55.2% of cases, while adding measured pairwise interactions raises this rate to 68.6%.

  3. The paper evaluates several frame selection methods. Section 6 tests the stability of the findings by changing frame order, candidate positions, surrounding frames, and frame budget.

Critical weaknesses
  1. Limited new insights and no real progress toward the research goal: The paper uses many experiments to confirm intuitive findings that evidence frames are more useful, and ranking frames by individual scores can miss the best combination. However, it provides little further explanation of these findings. The analysis does not clearly identify the specific causes of selection failures. It also does not propose and test a practical strategy to address these causes. This is my main concern with the paper.

  2. The experiments miss the core issue, and the analysis is unconvincing: A large, even excessive, number of experiments repeatedly confirm the advantage of evidence frames over outside frames. In contrast, the four-candidate experiment directly studies combination selection but receives little analysis. Figure 3 (right) compares the absolute interaction of evidence-frame pairs and outside-frame pairs. However, the frames in the two groups already differ. The individual and joint gains of outside frames are both close to zero, so their interaction is naturally close to zero as well. The paper does not clearly explain why comparing these values is meaningful. Section 6 and Table 2 then repeatedly verify this gap through stress tests under various settings. The extensive analysis with unclear value is another major reason for my recommendation to reject the paper.

Overall, the paper has a reasonable motivation, but it provides no substantial analysis of the causes and no solution. And the writing is also difficult to understand. I recommend clear rejection.

Questions for authors
  1. The four-candidate experiment relies on scores for the correct answer. How can these scores be obtained when the answer is unknown?

  2. Methods such as DPP and MDP3 in Section 5.3 select frames using embedding-based scores. How do their frame rankings relate to rankings based on correct-answer scores? Can the preceding analysis help these methods select frames?

Reviewer_McdaOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies whether a frame that is useful to a VidLM remains useful when combined with other frames. Most frame selection methods score each frame independently and simply keep the top K, ignoring how frames interact. The authors measure a frame’s utility by how much adding or swapping it into a fixed 8-frame input changes the model’s log odds of the correct answer against all other answers. Across 6 open VLMs and 3 grounded benchmarks, they find four main results: (1) annotated evidence frames are more useful than other frames in 17 of 18 settings; (2) evidence-frame pairs interact more strongly than pairs of outside frames in all 18 settings, although much of this difference is explained by their stronger individual contributions; (3) third- and fourth-order effects account for about a quarter of the total interaction strength; and (4) using oracle-measured pairwise interactions to select three out of four candidate frames reduces selection regret by 58%, while simple corrections based on individual frame scores do not help. Comparing training-free selection methods, a plain DPP achieves the highest mean log odds. However, its accuracy advantage is not statistically significant, and no method performs best across all settings.

Critical strengths
  • Looking at frame selection as a set function and asking whether individual frame scores are enough to explain the value of a set is a simple and useful way to study the problem.

  • The experiments are carefully designed and conducted at a reasonably large scale. Specifically, (1) the authors change only the frames being tested while keeping everything else fixed, (2) they repeat this across six models and three benchmarks, with about a million model calls; and (3) they also use statistical tests and run many additional checks, making the core findings fairly convincing.

Critical weaknesses
  • The 18/18 result does not show that individual ranking is inadequate. The fact that frames interact do not necessarily mean top-K selection fails. Frames can interact strongly, yet ranking them individually can still select the best set. Furthermore, the gold-vs-rest log-odds can show interactions even when each frame affects each answer option independently. The paper acknowledges this issue but does not measure how much of the observed interaction comes from the metric itself.

  • Evidence frames are often sampled close together in time, while outside frames are spread across the video. Since the paper does not measure or control for visual similarity, stronger interactions between evidence frames may simply reflect redundancy between similar frames.

  • The paper does not provide any solution. It recommends DPP, but it is an existing selection method whose accuracy improvement is not even statistically significant. It also does not show that DPP benefits from the interactions measured in the paper. The proposed idea of selecting a shortlist and then refining it is just mentioned, but not tested.

  • The main experiments use only 8 frames. The gap already shrinks at 16 frames. Most recent VidLMs can process more than 16 frames. So, a more realistic settings with 32 to hundreds of frames are not tested.

Questions for authors
  • How are the evidence frames selected? For e.g., are they selected randomly from the evidence window or sampled at uniform time interval? What happens if a question has multiple evidence intervals?
  • How are the outside/outside pairs selected?
  • Could strong interaction between evidence frames be a result of them being visually similar?
Reviewer_SKx5OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

Multiple-choice video question answering systems often select frames by scoring each independently for relevance, change, or salience. The authors ask "Does a frame retain its value beside the other selected frames?" To answer this question, the authors define individual utility and pair interaction metrics to measure the contributions of individual frames and their interaction effects. By conducting experiments using different models and benchmarks, the authors find that (1) evidence is useful (2) frames interact (3) independent ranking can miss the best combination.

Critical strengths

The authors conduct comprehensive experiments using multiple models and benchmarks.

Critical weaknesses
  1. The paper is difficult to follow, and its writing needs substantial improvement. For example, including a few sample questions in the Introduction would help readers better understand the task and motivation. Technical terms should be defined when they are first introduced, and overly long sentences should be avoided. Currently, the presentation quality does not meet the standards of ICLR.

  2. The authors do not investigate how interaction effects vary across different types of questions. For example, some questions can be answered using a single frame, for which interaction effects may be relatively small. In contrast, other questions require information from multiple frames and may exhibit stronger interactions. By aggregating results across these question types, the paper makes it difficult to determine what drives the observed interaction effects.

Questions for authors
  1. To what extent are interaction effects influenced by different types of questions? Could this affect the conclusions drawn in this paper?

  2. For individual utility, why is each candidate frame inserted at the fourth position? This fixed placement does not reflect real-world scenarios, where evidence frames may appear at different positions in the input sequence. The authors should justify this choice and investigate whether the individual utility estimates are sensitive to the insertion position.

  3. The paper acknowledges that frame interactions are context- and position-dependent, but primarily reports aggregate interaction magnitudes. It remains unclear how stable the sign, magnitude, and relative ranking of individual pair interactions are across different contexts and presentation positions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.