acceptodds
Withdrawn from ICLR 2027

Stable Curves, Unstable Items: Item-Level Scaling Heterogeneity in Video LLMs

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

More visual input often improves average accuracy in video language models. However, aggregate gains do not reveal whether models preserve answers they already get right. We present a paired evaluation protocol that tracks the same questions across visual budgets within frozen models. Our study covers four models and four multiple-choice benchmark splits on a common budget grid, including text-only input. On these grids, – of questions exhibit at least one correct-to-incorrect transition as the budget increases. Choosing the best budget for each question in hindsight yields – percentage points higher accuracy than the best fixed budget. This gap reveals complementary successes that no single budget captures. We release over 120,000 per-item evaluation records and analysis code to support reproducible evaluation of visual scaling. Our findings show that larger visual budgets change which questions are answered correctly, not simply how many.

Primary area:
datasets and benchmarks
Keywords:
Video LLMs, Scaling Behavior, Temporal Reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.80 over 5 reviews, above 27% of the others.
Reviewer_jaoROpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether increasing the visual input budget preserves the individual questions that a video language model already answers correctly. It introduces a paired evaluation protocol that tracks the same questions across input configurations while keeping the model fixed, separating correct-to-incorrect transitions from improvements. Across four main models and four multiple-choice benchmark splits, the shared text-only/16/32/64-frame grid yields confusion rates of 12.5–25.5% and retrospective oracle gains of 8.8–18.9 percentage points over the best fixed configuration. Additional experiments examine another model family, open-ended generation, dialogue, input sampling, spatial resolution, and adaptive inference. The paper also provides extensive per-item records and analysis code. Its main contribution is an empirical characterization of item-level variability that aggregate scaling curves can conceal.

Critical strengths

1.The paper identifies a meaningful limitation of aggregate accuracy: improvements can coexist with losses on previously correct items. Tracking paired outcomes provides a straightforward and interpretable way to expose this behavior. The distinction between rescued and harmed items is particularly useful for understanding the examples in Figures 1 and 2.

2.The common-grid comparison covers 16 model–split combinations, and the additional model, generation, and dialogue evaluations broaden the evidence. The repeatability checks, raw-versus-cached comparisons, and fixed-budget sampling and resolution studies help establish that the central observation is reproducible across several evaluated settings (Sections 3–5).

3.The authors carefully distinguish retrospective oracle headroom from achievable improvements, requested frame budgets from computational cost, and label-informed diagnostics from deployable selectors. Section 6 also shows why accounting for all inference probes can change the apparent efficiency of a cascade. The released per-item records, protocol documentation, and analysis scripts make the study useful for subsequent evaluation and adaptive-inference research.

Critical weaknesses

1.The empirical characterization is useful, but the contribution remains primarily descriptive. The failure analysis does not identify a reliable explanation for the observed instability, and the adaptive-inference analysis mainly establishes opportunities and limitations. Relative to the sampling and scaling studies discussed in Section 7, it would help to state more precisely which substantive conclusion becomes possible through the proposed paired analysis and how it should change evaluation practice.

2.The main grid includes text-only input, while increasing the requested frame count can also change which frames are selected. Consequently, “visual confusion” includes both losses of text-only successes and regressions between visual configurations with potentially different evidence. These choices are disclosed, but the headline figures do not clearly separate their contributions. This limits how directly the results can be interpreted as a failure to benefit from additional visual information (Section 2 and Section 5.2).

3.The primary generation metrics, Token-F1 and ROUGE-L, measure reference overlap and can change because of wording as well as answer quality. The authors appropriately distinguish these score regressions from binary correctness and provide a secondary rubric judge in Appendix B.2 and the supplement. However, the relationship between the judge-based findings and the main overlap-based regressions is not sufficiently summarized, making the broader implications for semantic answer quality harder to assess.

Questions for authors

1.What do the authors consider the most substantive insight established by the paired analysis beyond sensitivity to input configuration? Could you identify the most important concrete change that this finding should motivate in video-model evaluation?

2.Using the existing predictions, could you briefly summarize how the confusion rates and oracle gaps change when the text-only configuration is excluded? Given that sampled frame sets need not be nested, which conclusions should be understood specifically as visual-budget effects, and which concern sensitivity to evidence selection more generally?

3.Could you summarize the existing rubric-judge results and explain whether they support both oracle complementarity and answer-quality regressions? In particular, how closely do they support the interpretation of the regressions reported under Token-F1 and ROUGE-L?

Reviewer_jeukOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Increasing the visual input is generally expected to improve model performance. This paper compares the performance of video language models across different visual budgets and finds that average accuracy can mask item-level changes. The authors investigate this phenomenon through experiments across multiple models and benchmarks.

Critical strengths
  1. The question is practically important in VLLM evaluation, with aggregate accuracy can hide item-level performance changes.
  2. The paper provides extensive experimental coverage across multiple models, benchmarks, and visual input configurations.
Critical weaknesses
  1. Frame-count effects are not sufficiently isolated from sampling effects. Different budgets may sample entirely different frames, making it difficult to attribute item-level regressions specifically to changes in frame count.
  2. The robustness of item-level regressions remains unclear. Repeating the experiments with multiple random seeds at each budget would help determine whether the observed correct-to-incorrect transitions are consistent.
  3. The analysis provides limited mechanistic insights. Video and question features fail to explain regressions robustly, limiting the guidance for model improvement.
Questions for authors
  1. If higher-budget inputs retain all frames from lower-budget inputs and only add new frames, do the regressions still occur?
  2. Across multiple sampling seeds, do the same items consistently change from correct at lower budgets to incorrect at higher budgets?
Reviewer_oEFUOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes an unamed evaluation procedure for quantifiably evaluating how additional visual input affects accuracy from baseline relative to the number originally answer questions correctly. The paper also shares its 120,000 record evaluation dataset and empirical findings over 16 model-split comparisons.

Critical strengths

The author's approach seems mathematically sound as presented in Eq 1.

Critical weaknesses

Research Quality: The findings are not particularly suprising or novel. I concede that this could in part be do to presentation, but variations in downstream accuracy are expected to vary with input. Measuring the harm to original answers is an interesting analytic tool... but doesn't warrant publication on its own and the research value of this tool is not communicated effectively.

Paper Writing: I do not mind that an LLM was used to assist in the writing, but many segments of the paper clearly seem to be verbatim outputs from LLM without human authorship. I present this not as grounds for rejection on it's own, in keeping with ICLR's LLM policy, but raise it as weakness in the paper's writing quality:

  • It relies heavily on informal LLM-style distinction statements being made over formal definitions (example [1]). These are ubitquitious throughout the paper and shallow.
  • The descriptions and writing is compact and uneccsarily difficult to approach throughout the paper.
  • The findings are not well framed but rather coldly and densely dumped for the reader to interpret without flow or clear takeaways - nor is there structure to this presentation for the reader.
  • The discussion of findings is poor and so is the positioning of the research itself with respect to related work, it reads as dump of statements and references.
  • Summary of findings in the introduction doesn't even report the takeaways from the results.

(1) - 029: "This distintion matters when..." [The previous sentence is not a technically distinction, the authors mean to say that there are two sets of answers: one which produces the higher overall accuracy and the original set of answers.]

As the paper currently stands, I strongly recommend rejection. It doesn't show the maturity required of a conference publication. If the research value can be communicated, and if the if the paper writing can presented appropriately, then I would consider changing my grade. However, this would require an almost entirely changed draft.

Questions for authors

Did the authors audit the AI generated code? Does their code include validation tests for this effect? Can the authors provide a better, human authored, summary of their takeaways and findings? I think this would be required inorder to adaquately present their findings in a conference-appropriate setting.

Reviewer_UGL7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper examines whether increasing the visual budget preserves answers that a frozen Video LLM already gets right. The authors track the same questions across budgets, measuring correct-to-incorrect transitions and the gap between an item oracle and the best fixed configuration. Across four main models and four MCQA subsets on a shared text/16/32/64-frame grid, they report 12.5-25.5% visual confusion and 8.8-18.9 percentage points of oracle headroom. They also examine generation scores, input protocols, and adaptive inference, and describe a release of over 120,000 per-item records.

Critical strengths

S1. The paired results show what the aggregate curves miss. On the Qwen2.5 short-video subset, moving from 128 to 256 frames changes 18 answers from wrong to correct and 23 from correct to wrong, despite only a 0.6-point accuracy decline. Similar patterns across models and datasets provide evidence that this is a recurring issue. Using a shared configuration grid also makes the oracle gaps easier to compare (Figures 1-2, Table 2).

S2. The cascade cost accounting is useful. The three-stage policy uses 87.4 frames on average at its final selected stage, but the total rises to 118.9 when the earlier passes are included. This substantially changes the apparent saving relative to fixed 128-frame inference. The paper also reports the slower offline latency estimate rather than treating fewer requested frames as evidence of a speedup (Section 6, Appendix D.3).

Critical weaknesses

W1. My main concern is how much the study adds beyond the finding that different frame samples lead to different answers. Lower-budget frames are not necessarily retained at higher budgets, so a correct-to-incorrect transition may result from losing relevant evidence. Section 5.1 already reports 18 such transitions when only the sampling policy changes at 64 frames. The reported answer changes are convincing, but this confound limits what can be concluded about increasing the input budget itself. A control that retains the original frames as the budget increases would help distinguish these explanations. It would also help to report the headline statistics without the text-only configuration (Sections 2 and 5, Appendix C.1).

W2. I give less weight to the generation results because both Token-F1 and ROUGE-L measure word overlap. A change in phrasing or answer length can lower both scores without making an answer less correct. The reported tables therefore do not show how often a score regression corresponds to a worse answer. Appendix B.2 mentions a Qwen2.5-VL-32B rubric judge, but its results are not presented in the PDF. Those results would be more informative here than another lexical metric (Section 4, Appendix B.2).

W3. The evaluated subsets need a clearer explanation. Video-MME v1 short and medium contain 900 questions each in the benchmark paper, whereas this study uses 786 and 639 without explaining which videos were excluded and why. For v2, Appendix A.1 states that the upstream revision and full exclusion record could not be recovered. These gaps matter for a paper that presents its evaluation records as a contribution: readers need to know how the subsets were constructed and whether the exclusions changed their composition (Table 1, Appendix A.1).

Questions for authors

Q1. Are results available with nested frame sets, so that every lower-budget frame is retained at the next budget? How do the main confusion rates and oracle gaps change when the text-only configuration is excluded?

Q2. What do the secondary rubric-judge results show for the generation score regressions? Do they confirm that the answers become less correct, or are many changes attributable to wording or length?

Q3. Which videos were excluded from the v1 short and medium subsets, and why? For v2, what source and selection information is available to reconstruct the evaluated subset?

Reviewer_gNwsOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper is an experiment report around the idea: giving more frames / visual tokens a VLM does not monotonically increase answer accuracy.

Method-wise, the paper does not train anything or changes any architecture. All evaluations are done using existing VLMs (Qwen2.5-VL-7B, Qwen3-VL-8B, InternVL3-8B, InternVL3.5-8B, and LLaVA-NeXTVideo-7B) with existing benchmarks (VideoMME v1/v2, MLVU, and AVSD) by asking the VLM questions and take statistics of accuracies.

The paper mentions "adaptive inference": build an eval set, gradually increase number of frames in the context until the eval set performance starts to decrease. Also several different ways of frame sampling: uniformly from video, random, dense start.

Critical strengths

The clarity of the paper is good.

I can follow the settings and metrics easily.

Critical weaknesses

The main issue is that the core message of this paper, "giving more frames / visual tokens a VLM does not monotonically increase answer accuracy," in my opinion is trivial even in the early days of LLM (when we tried not to throw more than 4k tokens to GPT4, because even though GPT4's context supported 32k tokens, haullucination started to pop up a lot with 4k+ tokens in kv cache).

The result is not surprising and does not provide new insight. In fact this phenomenon and inference strategy / frame selection have already been well examined.

Literature review and comparisons are missing. I recommend a literature research around the following works:

  • Qu, Tianyuan, et al. "Does your vision-language model get lost in the long video sampling dilemma?." 2025 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, 2025.
  • Buch, Shyamal, et al. "Flexible frame selection for efficient video reasoning." 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2025.
  • Sun, Hui, et al. "Mdp 3: a Training-Free Approach for List-Wise Frame Selection in Video-Llms." 2025 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, 2025.
  • Zhang, Shaojie, et al. "Q-frame: Query-aware frame selection and multi-resolution adaptation for video-llms." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2025.
Questions for authors

Can the authors compare against the literature above? What are the new discoveries? What are the new methods?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.