- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates whether increasing the visual input budget preserves the individual questions that a video language model already answers correctly. It introduces a paired evaluation protocol that tracks the same questions across input configurations while keeping the model fixed, separating correct-to-incorrect transitions from improvements. Across four main models and four multiple-choice benchmark splits, the shared text-only/16/32/64-frame grid yields confusion rates of 12.5–25.5% and retrospective oracle gains of 8.8–18.9 percentage points over the best fixed configuration. Additional experiments examine another model family, open-ended generation, dialogue, input sampling, spatial resolution, and adaptive inference. The paper also provides extensive per-item records and analysis code. Its main contribution is an empirical characterization of item-level variability that aggregate scaling curves can conceal.
Critical strengths
1.The paper identifies a meaningful limitation of aggregate accuracy: improvements can coexist with losses on previously correct items. Tracking paired outcomes provides a straightforward and interpretable way to expose this behavior. The distinction between rescued and harmed items is particularly useful for understanding the examples in Figures 1 and 2.
2.The common-grid comparison covers 16 model–split combinations, and the additional model, generation, and dialogue evaluations broaden the evidence. The repeatability checks, raw-versus-cached comparisons, and fixed-budget sampling and resolution studies help establish that the central observation is reproducible across several evaluated settings (Sections 3–5).
3.The authors carefully distinguish retrospective oracle headroom from achievable improvements, requested frame budgets from computational cost, and label-informed diagnostics from deployable selectors. Section 6 also shows why accounting for all inference probes can change the apparent efficiency of a cascade. The released per-item records, protocol documentation, and analysis scripts make the study useful for subsequent evaluation and adaptive-inference research.
Critical weaknesses
1.The empirical characterization is useful, but the contribution remains primarily descriptive. The failure analysis does not identify a reliable explanation for the observed instability, and the adaptive-inference analysis mainly establishes opportunities and limitations. Relative to the sampling and scaling studies discussed in Section 7, it would help to state more precisely which substantive conclusion becomes possible through the proposed paired analysis and how it should change evaluation practice.
2.The main grid includes text-only input, while increasing the requested frame count can also change which frames are selected. Consequently, “visual confusion” includes both losses of text-only successes and regressions between visual configurations with potentially different evidence. These choices are disclosed, but the headline figures do not clearly separate their contributions. This limits how directly the results can be interpreted as a failure to benefit from additional visual information (Section 2 and Section 5.2).
3.The primary generation metrics, Token-F1 and ROUGE-L, measure reference overlap and can change because of wording as well as answer quality. The authors appropriately distinguish these score regressions from binary correctness and provide a secondary rubric judge in Appendix B.2 and the supplement. However, the relationship between the judge-based findings and the main overlap-based regressions is not sufficiently summarized, making the broader implications for semantic answer quality harder to assess.
Questions for authors
1.What do the authors consider the most substantive insight established by the paired analysis beyond sensitivity to input configuration? Could you identify the most important concrete change that this finding should motivate in video-model evaluation?
2.Using the existing predictions, could you briefly summarize how the confusion rates and oracle gaps change when the text-only configuration is excluded? Given that sampled frame sets need not be nested, which conclusions should be understood specifically as visual-budget effects, and which concern sensitivity to evidence selection more generally?
3.Could you summarize the existing rubric-judge results and explain whether they support both oracle complementarity and answer-quality regressions? In particular, how closely do they support the interpretation of the regressions reported under Token-F1 and ROUGE-L?