acceptodds
Withdrawn from ICLR 2027

CASE: Cost-Aware Stopping for Efficient Long-Video Agents

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Long-video agents can actively gather question-relevant evidence, but they typically leave a central decision implicit: when has the agent seen enough to answer? We propose CASE, a plug-in termination framework that frames this decision as policy-conditioned sequential stopping. At each causal checkpoint, CASE combines an auxiliary multiple-choice assessment of accumulated evidence with the host agent's execution state. From complete native trajectories, we construct a cost-aware target that compares answering now with stopping later along the same search path, accounting jointly for answer correctness and the full cost of continued reasoning. A lightweight Ridge regressor learns this decision gap and produces STOP/CONTINUE decisions. We evaluate three vision-language models with VideoSeek and AVP. On Video-MME, end-to-end accuracy changes by percentage points on average while CASE reduces model-token use by 53.63%. The same frozen policies then transfer zero-shot to LongVideoBench and MLVU, with end-to-end accuracy changes of and points while saving 58.78% and 51.28% of model tokens, respectively. Across all agent–model–benchmark combinations, CASE attains the highest accuracy–efficiency Pareto-frontier coverage among the compared stopping methods (83.3%) at the selected operating points. Online execution preserves this favorable accuracy–efficiency trade-off and additionally reduces measured runtime by 54.1% on average. CASE provides a plug-in termination framework for long-video reasoning agents, enabling them to decide when further evidence acquisition is no longer worthwhile.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Video Agents, Cost-Aware Inference, Token Efficiency

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_AVGNOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes CASE, a cost-aware stopping framework for long-video agents that learns when to terminate evidence acquisition. It trains a lightweight Ridge regressor using answer-option probabilities, agent execution states, and cost-aware supervision derived from complete trajectories. Experiments across two video agents, three VLMs, and three benchmarks demonstrate substantial token savings with competitive accuracy, including encouraging zero-shot cross-benchmark transfer results.

Critical strengths
  1. Practical significance and clear motivation. The paper addresses an important and practical problem in long-video agents: deciding when further evidence acquisition is no longer worth its computational cost. The motivation is clear, and the proposed cost-aware stopping formulation is intuitive and effective, offering a principled alternative to heuristic stopping rules based on confidence or fixed budgets.

  2. Strong efficiency gains and cross-benchmark generalization. CASE achieves substantial computational savings across multiple video agents and VLM backbones without modifying their underlying search policies. The zero-shot transfer results on LongVideoBench and MLVU are encouraging, suggesting that the learned stopping policies can generalize across benchmark distributions. The online experiments further demonstrate meaningful reductions in actual inference latency.

Critical weaknesses
  1. Limited applicability beyond closed-ended video QA. The proposed method is designed specifically for multiple-choice video QA, relying on answer-option probabilities and a constrained prefix finalizer. It is unclear how this framework could be extended to open-ended video understanding or more general video-agent tasks, where answer correctness and evidence sufficiency are harder to assess. This limits the general applicability of the proposed approach.

  2. Limited advantage under matched computation budgets. Although CASE achieves strong Pareto-frontier coverage at its selected operating points, the common-budget comparisons in Table 17 show little improvement over simpler baselines such as tuned token budgets and Forward-Q. In several settings, these baselines achieve comparable or better accuracy with similar computational savings. This raises questions about how much benefit the proposed cost-aware learning objective provides beyond selecting a more aggressive stopping threshold.

Questions for authors
  • Benefit over simpler stopping baselines: Table 17 shows that CASE does not consistently outperform simpler approaches, particularly tuned token budgets, under comparable computational budgets. Could the authors clarify what practical advantage the proposed cost-aware learning objective provides beyond threshold tuning? An analysis of when CASE makes better stopping decisions than these simpler baselines would help establish its contribution.
Reviewer_WzRbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses early termination in tool-augmented long-video QA, framing evidence collection as an optimal stopping problem over fixed agent trajectories. The authors introduce CASE, a lightweight plug-in controller that evaluates a STOP/CONTINUE objective prior to each candidate tool invocation without altering the host agent or base VLM parameters. At each decision point, an 18-dimensional state vector is formed by concatenating MC-probe option likelihoods, accumulated token expenditure, search progress, and tool category. Training labels are derived retrospectively from full rollouts by computing the decision gap between immediate termination loss (classification error plus normalized token cost) and the downstream trajectory optimum. This target is modeled via weighted Ridge regression, where the cost trade-off and decision threshold are tuned on a shared calibration split. Upon early halting, answer generation is delegated to a constrained prefix finalizer conditioned on the existing evidence cache. Evaluated across two agent frameworks (VideoSeek, AVP) and three open-source VLMs, controllers calibrated on Video-MME transfer zero-shot to LongVideoBench and MLVU. The method yields 51% to 59% token savings alongside marginal to moderate accuracy gains (+0.67, +3.38, and +4.58 points), defining the empirical Pareto frontier in 15 of 18 evaluated settings and cutting wall-clock runtime by 54.1%. The appendix supplements these findings with budget-matched controls, trajectory seed robustness checks, and an ablation isolating prefix-finalizer gains from termination mechanics.

Critical strengths
  1. The paper isolates termination as a separate decision in long-video agents and treats it as its own module. This framing is easy to adopt, since CASE works with existing hosts and does not change the VLM or the search policy.
  2. The controller is a linear model trained on only 150 questions per configuration, and supervision costs a few million tokens.
  3. The experiments and analysis are detailed and solid.
Critical weaknesses
  1. The 18-dimensional state is manually chosen, and the tool-type block is constant ("none") for AVP, so 5 of the 18 dimensions carry no information there. Each agent–model pair needs its own adapter and Ridge model, and the learned coefficients differ substantially across pairs. Because the execution-state ablation yields performance competitive with the full model, the empirical necessity of the answer-support features remains unproven.
  2. The transfer targets (LongVideoBench, MLVU) are multiple-choice long-video QA benchmarks similar to Video-MME, and the policy is never transferred across hosts or VLMs. Every new agent–model combination therefore needs its own native trajectories and refitting. Results are also sensitive to the train/calibration split even in-domain.
  3. In Table 17, CASE fails to beat Forward-Q or TASR-Video under matched budgets, and even trails a tuned-budget baseline at specific targets. The reported PFC gain is largely an artifact of the $\ge$55% saving cutoff, which disproportionately favors CASE's aggressive halting. The authors should plot full accuracy-efficiency curves across the entire budget spectrum for all methods.
  4. The paper should discuss recent long-video stopping and evidence-sufficiency methods [1-3] and RL-trained video agents that learn when to stop [4-5].

[1] REVEAL: A Rubric-Guided Agent for Explicit Evidence Sufficiency Verificationin Long-Video Question Answering, arXiv 2026 [2] When and Where to Look: Adaptive Visual Evidence Scheduling for Efficient Long Video Understanding, arXiv 2026 [3] TRACE: Temporal Retrieval with Anchored and Convergent Evidence for Long-Horizon Video Understanding, EMNLP 2026 [4] Video-MTR: Reinforced Multi-Turn Reasoning for Long Video Understanding, ICML 2026 [5] SAGE: Training Smart Any-Horizon Agents for Long Video Reasoning with Reinforcement Learning, CVPR 2026

Questions for authors
  1. Appendix C.4 attributes +4.80 pp of CASE’s +2.88 pp to the constrained finalizer and -1.92 pp to stopping itself. What causes the native terminal answer to fall about 4.8 pp below the same constrained finalizer applied to the full trajectory? Is it unparseable or invalid outputs, formatting failures, or something else?
  2. Table 9 reports the finalizer and stopping effects only as averages over 18 configurations. Could the authors provide this split for each cell?
Reviewer_z4owOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes CASE (Cost-Aware Stopping for Efficient Long-Video Agents), a lightweight, plug-in termination framework designed to decide when a tool-guided long-video question-answering agent has gathered sufficient evidence to stop searching and answer. While leaving the host agent’s evidence-acquisition policy ("where to seek") and the underlying vision-language model (VLM) unchanged, CASE formulates the termination decision as a policy-conditioned sequential stopping problem under state-dependent, heterogeneous computation costs.

Critical strengths

1.Well-Motivated, Lightweight, and Plug-and-Play Design: The paper addresses a practical bottleneck in agentic long-video understanding by cleanly decoupling when to stop from where to seek. It requires no fine-tuning of the underlying VLM, preserves the host agent's native search behavior.

  1. Comprehensive Evaluation: The empirical validation is thorough, spanning 18 configurations alongside real online execution tests that confirm model-token savings (54.6%) translate directly into wall-clock runtime reductions (54.1%).
Critical weaknesses

1.Restricted Applicability to Closed-Set Multiple-Choice QA: The proposed framework is tightly coupled to multiple-choice question answering (MCQ) and cannot be directly applied to open-ended video QA or free-form video reasoning tasks.

2.Confounded Accuracy Gains, Unverified Stopping Grounding, and Noisy Hindsight Supervision: While the main text highlights an average end-to-end accuracy improvement of +2.88 pp over Native execution, the attribution analysis in Appendix C.4 (Tables 9 and 10) reveals that this gain is primarily driven by the constrained prefix finalizer. The paper does not verify whether CASE actually stops right after locating ground-truth temporal evidence (e.g., since 80%–100% of stops in several configurations occur at Checkpoint 0 or 1). Besides, the model has a non-trivial chance of randomly guessing the correct option in 4- or 5-way MCQs even without relevant visual evidence, a lucky guess at any later checkpoint sets the future error term to zero. This introduces optimism bias and label noise into the regression targets, potentially misleading the controller about the true value of continued search.

3.Inclusion of Untouched Short Videos in Metrics and Missing Ultra-Long Benchmark (LVBench): By design, CASE bypasses Short videos on Video-MME and falls back entirely to native execution. Including these untouched short videos when reporting overall accuracy changes and token savings dilutes the true impact and failure modes of the stopping controller on the trajectories where it actually intervenes; evaluation metrics should be isolated and reported specifically on the active Medium and Long subsets. Furthermore, since CASE targets long-horizon video reasoning, evaluating on LVBench would provide a much stronger test of the method's scalability and effectiveness.

Questions for authors

1.Restricted Applicability to Closed-Set Multiple-Choice QA: The proposed framework is tightly coupled to multiple-choice question answering (MCQ) and cannot be directly applied to open-ended video QA or free-form video reasoning tasks.

2.Confounded Accuracy Gains: The main text highlights an average end-to-end accuracy improvement of +2.88 pp over Native execution, the attribution analysis in Appendix C.4 (Tables 9 and 10) reveals that this gain is primarily driven by the constrained prefix finalizer.

3.Unverified Stopping Grounding: The paper does not verify whether CASE actually stops right after locating ground-truth temporal evidence .

4.Noisy Hindsight Supervision: The model has a non-trivial chance of randomly guessing the correct option in 4- or 5-way MCQs even without relevant visual evidence, a lucky guess at any later checkpoint sets the future error term to zero. This introduces optimism bias and label noise into the regression targets, potentially misleading the controller about the true value of continued search.

Reviewer_55iuOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

Addressing the implicit decision-making challenge faced by long-video reasoning agents, specifically, determining when sufficient evidence has been gathered to formulate an answer, this paper introduces CASE, a plug-and-play termination framework. CASE models this as a sequential stopping problem conditioned on the agent's policy. At each "causal checkpoint," it integrates an auxiliary multiple-choice evaluation of the accumulated evidence with the host agent's current execution state. By leveraging full, native execution trajectories, it constructs a cost-aware objective that balances answer accuracy against the total cost of continued reasoning, training a lightweight Ridge regressor to output a "STOP" or "CONTINUE" decision. The authors evaluate CASE across three vision-language models using the VideoSeek and AVP datasets, reporting on accuracy, token consumption, and runtime performance, including zero-shot transfer to LongVideoBench and MLVU, as well as online execution.

Critical strengths

As a termination framework, CASE can be integrated into existing long-video reasoning agents without requiring modifications to their core search processes.

The cost-aware objective accounts for both answer accuracy and the full cost of further reasoning. By employing a lightweight regressor to learn the stopping decision, the framework achieves a balance between decision quality and computational overhead.

Experiments across multiple models and benchmarks demonstrate significant reductions in token usage, often while maintaining or even improving accuracy, alongside strong zero-shot transfer capabilities, a favorable Pareto frontier, and reduced online runtime.

Critical weaknesses

The rule for choosing settings requires at least 55% savings and then picks the setting with the most savings within a 2-point margin. This may push results toward settings that favor big savings.

Even with the same final step, CASE is 1.92 percentage points less accurate than the native (unmodified) agent. This suggests its gains may depend partly on the exact path used to reach the answer.

Adjusting for different costs mainly moves the setting toward bigger savings, not toward better accuracy.

The stopping policy is trained on only 50 videos. This limits how much data is used for supervision and calibration.

With the same fixed controller, savings vary by about 3.59 percentage points across different trajectory seeds. This shows it is still sensitive to the random nature of the host agent’s execution paths.

Questions for authors

Please refer to the weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.