acceptodds
Under review as a conference paper at ICLR 2027

Stop Early, Spend Less: Equal-Accuracy Frame-Budget Saving via Agreement-Gated Adaptive Sampling in Video Language Models

Abstract

Video large language models (VLMs) answer questions over long videos by sampling a fixed number of frames. Practitioners pick this frame budget a priori, yet the accuracy-vs-frames curve saturates at a domain-dependent point: beyond it, extra frames yield no accuracy but multiply compute, and can even hurt. We show that a simple, training-free agreement-gated rule—answer with a few frames, and if a slightly larger sample agrees, stop early; otherwise escalate—matches the accuracy of a strong static-uniform baseline at its best global budget while spending far fewer frames. On Video-MME with five VLMs spanning three families and three scales (Qwen2.5-VL-7B; Qwen3-VL-8B; InternVL3-8B; InternVL3.5-8B; InternVL3.5-4B), STOP-EARLY is statistically non-inferior to static sampling at 24 frames (TOST, margin ±3 percentage points) while using approximately 41–48% fewer frames on three of the five models (the exception is the model with the steepest short-video curve). Across five video-QA settings (three Video-MME splits, TempCompass, MVBench), the benefit is a monotone function of the static curve's slope: the strong form (near-perfect rank correlation, Spearman rho at most -0.9) holds on three of the five models, and the weaker sign-level form (a negative slope predicts a free lunch) holds on all five. Where the curve is flat or declining, STOP-EARLY saves 45–54% of frames at equal or better accuracy (on TempCompass, +4.0 percentage points at -54% frames, and up to +5.5 percentage points on other models); where it rises steeply, the fixed high budget is genuinely needed—a predictable scope boundary. Since the optimal fixed budget drifts across domains and is unknown at deployment, a fixed budget tuned on one domain incurs regret on another, while STOP-EARLY tracks the per-domain optimum online. A rate-matched random gate fails non-inferiority, isolating that the gain comes from knowing when to stop. We release all code, per-item results, and analysis.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
video language models, adaptive inference, frame sampling, efficient multimodal models, early stopping

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_KwPgOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

This paper investigates the computational redundancy associated with fixed frame budgets in video large language models and proposes STOP-EARLY, a training-free adaptive sampling method. Specifically, the authors first perform inference on the same video using 6 and 10 frames. If the two answers agree, the answer obtained with 10 frames is returned directly; otherwise, the frame budget is increased to 24 for further inference. The authors further introduce the slope of the accuracy curve as a dataset-level criterion for assessing the method's applicability. Experiments are conducted on three benchmarks, and the results show that STOP-EARLY can maintain or improve accuracy while using fewer frames for certain models and tasks.

Critical strengths

The research content of this paper is relatively clear. It does not focus on temporal or semantic aspects, but directly determines whether a question can be answered with only a small number of frames. The implementation is simple and independent of model. The paper conducts several controlled experiments to validate the applicable scenarios of the method.

Critical weaknesses
  1. The method design appears highly subjective and simplistic. The paper uses answer consistency between 6-frame and 10-frame inputs as a stopping criterion. However, consistency between the two answers does not necessarily indicate that the model has acquired sufficient frames. On the contrary, it may also indicate that the model has not yet sampled enough informative frames. Some samples in existing benchmarks can be answered using only a small number of frames. Therefore, the experimental results appear to reflect statistical characteristics of the current benchmarks rather than demonstrate the effectiveness of the proposed method.
  2. Several key observations are not investigated in sufficient depth. For example, it remains unclear whether the observed down-flip phenomenon is caused by redundant information in the sampled frames, degraded model performance due to increased context length, or differences in the sampled content itself.
  3. The method may involve multiple independent model calls, such as separate inference runs with 6, 10, and 24 frames. Merely reporting the number of frames sampled is insufficient to demonstrate actual speedup in real-world deployment scenarios. A more comprehensive evaluation of inference latency and computational overhead is needed.
  4. The evaluation focuses primarily on multiple-choice questions, for which determining whether the answers generated from 6-frame and 10-frame inputs are consistent is relatively straightforward. However, real-world applications often involve open-ended question answering, where determining the consistency between responses generated from different numbers of frames can be substantially more challenging. The paper does not adequately address how answer consistency is assessed in such settings.
Questions for authors
  1. Short clips continue to benefit from increasing the frame budget to 24 frames, whereas TempCompass and long-video Video-MME plateau or decline earlier. The finding that using fewer frames can yield better performance on long videos appears counterintuitive and warrants further investigation.
  2. When the model obtains a_lo and a_mid using 6 and 10 frames, respectively, why is the total cost not counted as 16 frames, even when the two answers are consistent?
  3. Why were the specific frame budgets of 6, 10, and 24 selected? Are these budget settings generalizable across different models?
  4. The non-inferiority margin is set to ±3 percentage points. However, a performance difference of 3 percentage points is not negligible. Would the paper's conclusions still hold if a narrower margin were adopted?
Reviewer_cfYTOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes STOP-EARLY, a training-free adaptive frame-sampling method that uses answer agreement to decide whether further sampling is necessary. It evaluates five video language models across Video-MME, TempCompass, and MVBench, using non-inferiority testing to assess accuracy preservation. The paper reports substantial frame savings and identifies a relationship between sampling efficiency and the accuracy-vs-frames curve.

Critical strengths
  • Simple and training-free adaptive sampling method.
  • Evaluation across multiple models and benchmarks.
  • Statistical analysis with non-inferiority testing.
Critical weaknesses
  • The central efficiency claim relies on questionable cost accounting that does not fully account for multiple inference passes.

  • The experimental comparisons lack sufficiently strong matched-cost baselines, and several reported results are inconsistent.

  • The proposed slope-based explanation lacks rigorous validation, while the evaluation and reproducibility have important limitations.

Questions for authors
  1. The reported frame savings appear to exclude substantial inference costs. Algorithm 1 requires separate evaluations at 6 and 10 frames, followed by another evaluation at 24 frames when the answers disagree. However, the reported cost counts only 10 frames on agreement and 24 frames on disagreement. Even the paper's equation, 10+(1−π)×2410+(1-\pi)\times24, gives approximately 16.5 frames when π=0.73\pi=0.73, rather than the reported 13.5 frames. Counting all inference passes gives approximately 22.5 frames. How can the claimed 41--48% frame savings be justified? Please report total processed frames, visual tokens, and measured wall-clock latency, including all inference passes. The provided code also appears to sample the 6-frame and 10-frame inputs independently rather than using truly nested frames. Is any feature caching or incremental inference actually implemented?

  2. The comparison against static-24 may substantially overstate the advantage of adaptive sampling. Table 8 suggests that accuracy often saturates around 14--18 frames, and the provided results indicate that static-14 achieves comparable accuracy on several models while requiring only one inference pass. Why is static-24 treated as the primary baseline? Please provide comparisons against smaller fixed budgets under matched computational costs, as well as fixed budgets selected on a held-out validation set.

  3. Several reported results appear inconsistent. Table 12's accuracy differences relative to the per-domain optimum do not agree with Table 2, while the claimed optimal budget for Video-MME long appears to depend on the evaluated subset. Tables 6, 8, and 9 also appear inconsistent with other figures and the provided results. In addition, Figure 4 reports different frame-saving percentages from Table 2, and the down-flip rates differ across the Introduction, Section 5, and Table 6. Could the authors reconcile these discrepancies and regenerate the tables and figures using a consistent evaluation protocol?

  4. The proposed "slope law" may be partly determined by the evaluation construction. Since STOP-EARLY returns the 10-frame answer on most examples, its accuracy difference relative to static-24 is closely related to the difference between 10-frame and 24-frame accuracy. A negative correlation with the static accuracy slope may therefore arise mechanically. Can the authors demonstrate that the slope provides predictive information beyond simpler indicators, such as the 10-to-24-frame accuracy difference? Please also evaluate its predictive power on held-out settings and report statistical uncertainty for the Spearman correlations, which are calculated over relatively few points.

  5. The random-gate ablation does not fully establish the superiority of agreement-based stopping. Table 4 considers only one combined Video-MME setting, while the paper claims that random gating fails non-inferiority across models. Moreover, failing a non-inferiority test does not imply that the random gate is significantly worse than the proposed method. Could the authors directly compare the agreement gate and random gate using paired statistical tests across all five models and report confidence intervals for their performance differences?

  6. The evaluation protocol and released implementation require clarification. The provided code uses sampling procedures, output-token limits, and image resizing settings that differ from the paper, while several experiments appear to select the first available benchmark examples rather than representative random samples. Empty or unparsable answers may also lead to artificial agreement between low-budget predictions. Furthermore, some analysis scripts contain hard-coded local paths or depend on missing intermediate files. Could the authors clarify these implementation differences, describe how invalid answers are handled, and provide a self-contained reproduction package?

  7. The novelty and generalizability of the proposed stopping mechanism need stronger discussion. Agreement-based stopping is closely related to existing adaptive inference methods, including FrameExit (CVPR 2021), Adaptive-Consistency (EMNLP 2023), and Early-stopping Self-Consistency (ICLR 2024). How does STOP-EARLY differ technically from these approaches beyond applying agreement to frame-budget selection? Additionally, the evaluation primarily involves multiple-choice questions with short answers, where exact agreement is easy to measure. How would the method extend to open-ended video understanding tasks where semantically equivalent responses may have different wording?

[1] Ghodrati, Amir, Babak Ehteshami Bejnordi, and Amirhossein Habibian. "Frameexit: Conditional early exiting for efficient video recognition." CVPR. 2021.

[2] Aggarwal, Pranjal, Aman Madaan, and Yiming Yang. "Let’s sample step by step: Adaptive-consistency for efficient reasoning and coding with llms." EMNLP. 2023.

[3] Li, Yiwei, et al. "Escape sky-high cost: Early-stopping self-consistency for multi-step reasoning." ICLR. 2024.

  1. Minor: Several presentation and reference issues should be corrected. Figures 7, 9, and 14 contain labels associated with an apparently different method, including "Active probe (ours)" and uncertainty-based triggers, while Figure 8(b) has questionable axis values (x-axis values are all zero). Some citations also do not adequately support the accompanying claims, particularly those concerning the limitations of adaptive allocation and examples of early-exit methods. In addition, several references still cite arXiv versions despite the availability of formally published versions.
Reviewer_jwRAOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes STOP-EARLY, a training-free, per-question frame-budget rule for video LLMs: it asks the LLM to answer a question at a low frame budget (6) and a mid budget (10). If the two answers agree, return the mid-budget answer; if they disagree, escalate to a high budget (24).

Critical strengths
  1. Rigorous statistical evidences supporting the claim are presented in the paper.
  2. The limitations are mentioned honestly, like where the method fails (steeply rising curves) and admits the budget defaults are fixed.
Critical weaknesses
  1. The fixed three frame budgets (6, 10, 24) are applied uniformly across different benchmarks. How to choose this frame budget triplet for different datasets (say when very short and very long videos coexist in a dataset) is not discussed. Video duration, information content of the video, and time segment of interest are important factors in deciding frame budget. 6 frames for a video of <2 seconds might be an overkill, while 24 frames for a >100 minute video will be too lossy.
  2. Apart from the fixed/static budgets and some ablations (random gate, always-stop, never-stop), STOP-EARLY is not compared against the adaptive compute literature that the paper itself cites in its Related Work section.

Minor weakness:

  1. 'TOST' is used in the main text before defining the term.
Questions for authors
  1. Clarification: What is the exact down-flip rate? Line 439 says 5.2% while line 074 says 10%.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.