acceptodds
Withdrawn from ICLR 2027

Cardano: Training Streaming Video Language Models to Know When They Have Seen Enough

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Video language models (VLMs) answer a question about a video after processing all of it, so inference cost scales with video length rather than with the difficulty of the question. Processing the whole video is also not always beneficial, since additional video can overturn an answer that was already correct. The relevant question is therefore not how much video a model watches but when it should stop. We present Cardano, a training formulation for streaming VLMs that learns to answer questions and to decide when further observation would not change the answer. Rather than predicting the content of unseen video, Cardano predicts whether observing more video would change its current answer, which makes adaptive stopping a self-supervised problem: the target is obtained by comparing the model's answer on a partial video with its own answer on the full video, and requires no annotation or dataset-specific supervision. At inference the model reads the video window by window and, after each window, either commits to an answer or continues watching. A commit head of about forty parameters, fit once and deployed at its natural operating point with no threshold tuned on any dataset, outperforms a causal fixed budget at a matched number of frames on four of six benchmarks, with confidence intervals excluding zero, and transfers without refitting to a second architecture. Stopping preserves accuracy rather than improving it: with Qwen2.5-VL-7B, Cardano is within 1.5 points of its whole-video accuracy or above it on all six benchmarks while processing between one third and four fifths of each video. On FunQA and Video-MME, where training the answerer on video prefixes raises accuracy, Cardano exceeds the untrained backbone watching the whole video by 8.4 and 3.8 points. Code is available at https://anonymous.4open.science/r/cardano-vlm-404A.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
streaming video language models, video question answering, adaptive inference, early stopping, efficient video understanding, multimodal learning, video reasoning, vision-language models

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_A2FcOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes CARDANO, a training formulation for streaming video language models that learns when to stop watching by predicting whether further observation would change the current answer. It trains a lightweight commit head using self-supervised targets obtained by comparing the model's predictions on partial and full videos. Experiments across multiple video benchmarks show that CARDANO reduces the fraction of video watched while preserving accuracy and outperforming causal fixed-budget baselines at matched frame budgets.

Critical strengths
  • Relevant and pratical problem.
  • Lightweight and training-efficient mechanism.
  • Extensive benchmark evaluations
Critical weaknesses
  • The novelty over existing consistency-based early-exit methods is insufficiently established, and the independent contribution of the stopping mechanism remains unclear.

  • The experimental comparisons and efficiency analysis do not adequately isolate the benefits of adaptive stopping from answerer training and computational overhead.

  • Several methodological explanations, implementation inconsistencies, and reproducibility issues weaken confidence in the reported conclusions.

Questions for authors
  1. The novelty of CARDANO requires stronger differentiation from existing adaptive inference and consistency-based stopping methods. [1] already trains a lightweight classifier using agreement between intermediate and final predictions, which closely resembles CARDANO's self-supervised target. [2] also learns adaptive stopping decisions over video frames, while [3] addresses when sufficient evidence has arrived to answer questions in streaming videos. What is the fundamental technical contribution beyond transferring an existing consistency-based stopping objective from network depth to temporal observation? More direct comparisons and a clearer discussion of these closely related methods would strengthen the novelty claim.

  2. The evaluation does not sufficiently separate the contribution of answerer training from that of the commit head. The abstract reports improvements of 8.4 and 3.8 points over the untrained full-video backbone, but these gains may primarily result from prefix-based answerer training rather than adaptive stopping. The trained answerer's whole-video accuracy is not explicitly reported in the main tables, making it difficult to assess the actual accuracy cost or benefit of stopping. Could the authors add a "trained answerer, whole clip" baseline for every model and benchmark, and distinguish improvements from answerer fine-tuning and stopping? The comparison with tuned confidence thresholds also shows very small accuracy differences in several settings. A global confidence or patience policy fitted on the same training trajectories, with paired confidence intervals, would provide a more convincing comparison.

  3. The stopping policy's claim of requiring no threshold tuning deserves clarification. Although the commit head uses a fixed decision threshold of 0.5, its positive-class weight κ\kappa controls how aggressively the model stops, and the paper deploys a particular operating point selected from four trained heads. How was this configuration selected, and which data were used? Since class weighting can effectively shift the decision boundary, the distinction from threshold-based stopping is less clear than suggested. Furthermore, the self-supervised target is defined by agreement with the full-video prediction rather than correctness. Could the authors separate false commits caused by inaccurate head predictions from cases where the full-video prediction itself is incorrect, and report their frequencies across benchmarks?

  4. The released code and evaluation protocol contain substantial inconsistencies that limit reproducibility. The provided implementation has missing dependencies, including "stage_data" and "imagine_head", while "cardano/commit_head.py" uses "math.log" without importing "math". Several training settings also differ from the paper, including the default future-window offset, the direction of the KL objective, feature normalization, and tie-breaking. More importantly, the data loaders use custom evaluation partitions rather than the official benchmark test splits, and the MVBench evaluation includes videos overlapping with the answerer training data. The released materials also lack trained weights, trajectory dumps, and scripts needed to reproduce the main tables.

  5. Some methodological explanations and practical efficiency claims require stronger evidence. Section 3.2 infers limited predictability of future video information from a negative correlation between consecutive representation differences, but this does not establish that answer-relevant future information is unpredictable. The adjacent-window representation similarities reported in Appendix F also suggest a more nuanced interpretation. Could the authors provide stronger evidence supporting this mechanism and clarify the content-prediction baseline? Furthermore, Appendix J shows that CARDANO incurs 29-35% more wall-clock time than a fixed-budget method processing the same number of windows, and its relative runtime can exceed that of full-video inference. Since efficiency is a central motivation, comparisons at matched wall-clock latency, rather than only matched frames, would better establish the practical benefit.

  6. Several numerical, presentation, and bibliographic inconsistencies should be corrected. Tables 1 and 7 report slightly different CARDANO results for apparently identical settings, while some figure captions disagree with the corresponding visualizations or experimental descriptions. The qualitative examples also emphasize early successful commits without representing typical later stopping decisions. The bibliography contains several incorrect publication years and author lists, including the citation of STAR as a 2024 paper rather than its NeurIPS 2021 publication. Several references also retain outdated preprint information despite formal publication.

[1] Schuster, Tal, et al. "Confident Adaptive Language Modeling." NeurIPS. 2022.

[2] Ghodrati, Amir, Babak Ehteshami Bejnordi, and Amirhossein Habibian. "FrameExit: Conditional Early Exiting for Efficient Video Recognition." CVPR. 2021.

[3] Azad, Shehreen, Vibhav Vineet, and Yogesh S. Rawat. "StreamReady: Learning What to Answer and When in Long Streaming Videos." CVPR. 2026.

Reviewer_ABJ3OpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper aims to save the compute, and possibly improve accuracy, for VLM-based streaming video question answering through learning a neural network to decide when the decision can be made and no further frames are needed. The experiments show that in some cases this method can have better performance than let the VLM watch the full video.

Critical strengths

The research question is reasonable and has some significance.

Critical weaknesses
  1. The presentation is awkward and probably drafted by AI since there are many terms that smells like AI, but the authors did not include this usage of AI in the AI use statement.
  2. There are some improvement over the baseline methods, while the improvements are inconsistent and marginal.
  3. Details of competing methods are missing. What are the budget and K for fixed budget and first-K windows?
  4. No change of K experiment for first-K windows method, which might be a strong baseline.
Questions for authors

See weaknesses.

Reviewer_AnJiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary
  • To mitigate the computational overhead of processing full-length videos, this study introduces CARDANO, a framework that learns to trigger early stopping based on the predicted stability of the model's answer.
  • Experimental results demonstrate that CARDANO achieves performance highly competitive with full-video processing while significantly reducing cost.
  • As it requires no human labels or per-dataset thresholds, this method provides a highly scalable and adaptable solution for streaming video language models.
Critical strengths
  1. Clear Motivation and Well-Justified Problem Formulation: The paper provides a thoughtful analysis of the observation that sufficient information to answer a question may be available before the end of a video. Based on this insight, the research objective and method are well motivated, making the overall narrative clear and convincing. The discussion of computational costs also provides a balanced perspective on the practical benefits and limitations of the approach.

  2. Novel and Practical Learning Framework for Early Stopping: The formulation of the stopping objective and loss functions is interesting and original. In particular, the proposed approach eliminates the need for human annotations of stopping points, making the training framework more practical and potentially data-efficient. This is a valuable aspect of the method, especially for applications involving long videos.

Critical weaknesses
  1. Inconsistency between training objective and the goal of accuracy improvement - The paper defines the stopping target based on whether the model’s intermediate prediction matches its final-video prediction. This creates a logical inconsistency: if the model's final-video prediction is incorrect, the training objective will effectively supervise the model to converge on an incorrect answer early. The authors provide no mechanism to handle cases where an early answer is actually more accurate than the full-video response, potentially sacrificing potential gains in accuracy for the sake of mere stability.

  2. Potential for bias-driven predictions due to insufficient visual evidence - The training objective encourages the model to output a specific answer even when the video prefix may lack sufficient visual evidence. This likely forces the model to rely on dataset biases, statistical priors, or the language model's inherent hallucinations rather than genuine visual understanding. The current formulation fails to adequately address how the model distinguishes between "having enough information" and "making an educated guess based on priors," which undermines the reliability of the early stopping decision.

  3. The evaluation results, particularly in Table 2, demonstrate notable performance degradation compared to whole-clip or other method inference on several benchmarks. This suggests that the current trade-off between computational cost and accuracy is not always optimal, especially when the model prematurely commits to a stable but wrong prediction. The authors need to better justify this performance drop and clarify whether the competitive performance is consistently achievable or if it is highly sensitive to the specific characteristics of the dataset.

Questions for authors
  1. Stopping Criterion (Eq. 1): How does the proposed method handle cases where an intermediate prediction is correct but the full-video prediction is incorrect? Could using the full-video answer as the reference discourage beneficial early stopping?

  2. Training Objective (Eq. 3): How does the training objective account for video prefixes that do not yet contain sufficient visual evidence to answer the question? Could training on such prefixes encourage predictions based on dataset biases rather than visual evidence?

  3. Generalizability and Failure Cases (Table 2): What factors contribute to the performance degradation on TimeBlind, particularly the 3.7-point accuracy drop compared to full-video processing?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.