- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper investigates the computational redundancy associated with fixed frame budgets in video large language models and proposes STOP-EARLY, a training-free adaptive sampling method. Specifically, the authors first perform inference on the same video using 6 and 10 frames. If the two answers agree, the answer obtained with 10 frames is returned directly; otherwise, the frame budget is increased to 24 for further inference. The authors further introduce the slope of the accuracy curve as a dataset-level criterion for assessing the method's applicability. Experiments are conducted on three benchmarks, and the results show that STOP-EARLY can maintain or improve accuracy while using fewer frames for certain models and tasks.
Critical strengths
The research content of this paper is relatively clear. It does not focus on temporal or semantic aspects, but directly determines whether a question can be answered with only a small number of frames. The implementation is simple and independent of model. The paper conducts several controlled experiments to validate the applicable scenarios of the method.
Critical weaknesses
- The method design appears highly subjective and simplistic. The paper uses answer consistency between 6-frame and 10-frame inputs as a stopping criterion. However, consistency between the two answers does not necessarily indicate that the model has acquired sufficient frames. On the contrary, it may also indicate that the model has not yet sampled enough informative frames. Some samples in existing benchmarks can be answered using only a small number of frames. Therefore, the experimental results appear to reflect statistical characteristics of the current benchmarks rather than demonstrate the effectiveness of the proposed method.
- Several key observations are not investigated in sufficient depth. For example, it remains unclear whether the observed down-flip phenomenon is caused by redundant information in the sampled frames, degraded model performance due to increased context length, or differences in the sampled content itself.
- The method may involve multiple independent model calls, such as separate inference runs with 6, 10, and 24 frames. Merely reporting the number of frames sampled is insufficient to demonstrate actual speedup in real-world deployment scenarios. A more comprehensive evaluation of inference latency and computational overhead is needed.
- The evaluation focuses primarily on multiple-choice questions, for which determining whether the answers generated from 6-frame and 10-frame inputs are consistent is relatively straightforward. However, real-world applications often involve open-ended question answering, where determining the consistency between responses generated from different numbers of frames can be substantially more challenging. The paper does not adequately address how answer consistency is assessed in such settings.
Questions for authors
- Short clips continue to benefit from increasing the frame budget to 24 frames, whereas TempCompass and long-video Video-MME plateau or decline earlier. The finding that using fewer frames can yield better performance on long videos appears counterintuitive and warrants further investigation.
- When the model obtains a_lo and a_mid using 6 and 10 frames, respectively, why is the total cost not counted as 16 frames, even when the two answers are consistent?
- Why were the specific frame budgets of 6, 10, and 24 selected? Are these budget settings generalizable across different models?
- The non-inferiority margin is set to ±3 percentage points. However, a performance difference of 3 percentage points is not negligible. Would the paper's conclusions still hold if a narrower margin were adopted?