acceptodds
Under review as a conference paper at ICLR 2027

SAVE: Less is More in Agentic Video Reasoning

Abstract

Agentic video-reasoning frameworks iteratively explore a video, choosing what to observe based on the query and evidence gathered so far. Existing methods are typically evaluated with different backbones, preprocessing pipelines, and computational budgets, making it difficult to determine which components actually drive performance, how much they cost, and whether agentic inference is preferable to native video reasoning in the first place. We conduct the broadest controlled evaluation to date of a diverse set of recent agentic video reasoning systems across two strong contemporary backbones, with explicit token and monetary-cost accounting. Across Gemini 3 Flash and Gemma 4, most existing agentic frameworks fail to consistently improve over native video inference and often increase computational cost. We investigate two primary questions: (1) How much agentic framework complexity is necessary for effective video reasoning? (2) When, if at all, is agentic video reasoning preferable to just using the backbone directly? To answer the first question, we introduce a deliberately minimal Active Exploration Baseline and show that it remains competitive with substantially more elaborate agents. We then propose targeted extensions to this baseline, yielding SAVE (Simple Agentic Video Explorer). SAVE outperforms all evaluated prior agentic methods in five of six backbone–benchmark settings. On Minerva, SAVE reaches 66.4% accuracy with Gemini 3 Flash, comparable to 64.8% for native inference, while reducing inference cost by 60%, and exceeding the highest-accuracy evaluated prior agentic baseline by 9.9 percentage points with 79% lower cost. Motivated by this fact, we ask when agents are preferable to simply using the underlying backbone. Our results indicate that its advantage grows when dense processing would otherwise represent answer-relevant events too sparsely, particularly for long videos and temporally concentrated evidence. Overall, our results suggest that agentic video reasoning should favor simple frameworks over increasingly elaborate pipelines and focus primarily on long-form video, where selective evidence acquisition offers the clearest advantage over native inference.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Agentic Video Reasoning, Vision-Language Models, Keyframe Selection

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_VAgjOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Summary. The paper performs "the broadest controlled evaluation to date" of recent agentic video-reasoning systems "across two strong contemporary backbones, with explicit token and monetary-cost accounting." It unifies agents under a POMDP-inspired abstraction, reimplements six on Gemini 3 Flash and Gemma 4, and introduces "a deliberately minimal Active Exploration Baseline" and SAVE, which "lets the backbone query temporal ranges and adaptively allocates temporal and visual resolution within them." Contributions. (1) The "most comprehensive controlled evaluation to date… with explicit computational accounting, showing that most evaluated frameworks fail to improve over native inference while often substantially increasing cost." (2) A minimal baseline that "remains competitive with substantially more elaborate agents," plus SAVE, which "outperforms all prior agentic approaches in most settings at substantially lower computational cost." (3) A characterization of when "agentic exploration is most useful, namely when uniform processing would represent answer-relevant evidence too sparsely." Validation. Evaluated "across three benchmarks and two backbones," SAVE reaches "the highest performance among evaluated prior agentic methods in five of six settings," hitting "66.4% with Gemini 3 Flash, comparable to 64.8% for native inference," "while reducing monetary inference cost by approximately 60%." Significance uses video-clustered permutation and McNemar tests with Holm correction; its "advantage over a token-matched native baseline grows with video duration," and a synthetic experiment shows adaptive inspection helps most "when the temporal pattern… occupies only a small fraction of the video."

Critical strengths

The paper's greatest strength is its "less is more" stance, which runs against the field's trend of ever more elaborate agents and is especially interesting as backbones grow stronger. The authors show that "when paired with strong multimodal backbones, most existing agentic frameworks fail to consistently outperform native video inference," and that "a deliberately minimal Active Exploration Baseline… remains competitive with substantially more elaborate agents." They conclude that "agentic video reasoning should favor simple frameworks over increasingly elaborate pipelines."

This claim is backed by solid empirics: a controlled evaluation under shared backbones with explicit cost accounting, rigorous significance testing (video-clustered permutation and McNemar tests with Holm correction), and a clear characterization of when agents help, namely "when uniform processing would represent answer-relevant evidence too sparsely."

Critical weaknesses
  1. Agentic gains unproven. Table 2 shows richer temporal context helps. But there is no fixed non-agentic multi-stage sampler to rule out that no planning is needed. SAOS does not close this gap. It "restricts SAVE to a single conditioned localization" and is itself agentic. The paper never shows "agentic machinery is actually necessary."
  2. Overclaim on two backbones. It claims "agents do not consistently improve video reasoning performance, irrespective of the evaluated framework." Yet it tests only Gemini 3 Flash and Gemma 4. The method is backbone-agnostic. Stronger closed models like GPT-5.6 or GPT-6 Astra and an independent open family like Qwen 3.8 27B could be run. None are tested or justified.
  3. Modest uncontrolled results. On Gemini "the difference between SAVE and direct Gemini 3 Flash is not statistically significant (pHolm = 0.390)." SAVE also trails native on VSIBench with 57.1 versus 59.2. The one clear win on Gemma Minerva uses a native baseline capped at "at most 60 [frames] per call" while SAVE accesses the video over many rounds.
  4. Toy synthetic missing analysis. The synthetic videos repeat an event throughout and remove localization difficulty yet motivate long-video claims. For an efficiency paper "we do not include end-to-end latency." There is no per-question-type or failure-case analysis
Questions for authors
  1. Non-agentic control. Can a fixed coarse-to-fine sampler with no LLM decisions also reach 66.4% on Minerva under the same budget? SAOS does not answer this. It "restricts SAVE to a single conditioned localization" and is still agentic. Can you add a pure heuristic multi-stage sampling baseline?
  2. Cross-family backbones. The framework is backbone-agnostic at the API level. Why evaluate only Gemini 3 Flash and Gemma 4? Does "most existing agentic frameworks fail to consistently improve over native video inference" still hold on GPT-5.6 or Qwen 3.8 27B?
  3. Controlled native comparison. Native Gemma is capped at "at most 60" frames per call while SAVE accesses the video over many rounds. If native gets the same total frame budget does the Gemma Minerva win remain? On Gemini "the difference between SAVE and direct Gemini 3 Flash is not statistically significant (pHolm = 0.390)." How does this support the overall claim?
  4. Efficiency and regime. You ask when agents are preferable yet "we do not include end-to-end latency." Can you report latency call counts and per-question-type plus failure-case analysis to mark when SAVE underperforms native?
  5. As frontier models rapidly improve, both closed (e.g., 6astra) and open (e.g., Qwen3.8-27B) models offer ever stronger native long-context video understanding. Is the video reasoning task this paper targets still a real problem? More importantly, relative to simply plugging such strong models into a general-purpose base harness (e.g., Codex), where does your method's value and irreplaceability lie? Please provide a direct comparison against a "base harness + frontier model" setup.
Reviewer_Vcj3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper aims at evaluating agentic video reasoning. To achieve this, the authors conduct a controlled evaluation of video-agents by enforcing a common backbone. They define three baselines:

  1. Active Exploration Baseline, which chooses which video frame to inspect at each step and can elect to stop and output a final answer
  2. SAVE, which lets the agent select a video segment and adjusts the number of frames returned on its duration using an automatic scene detector (PySceneDetect), with higher-resolution frames provided for shorter segments.
  3. Scene-Aware Visual Query (SAOS), which is identical to SAVE but only allows a single visual query.

In the experiments, the authors replicate agentic visual reasoning methods with two backbones, Gemini 3 Flash and Gemma 4. They report results across three benchmarks Minerva, EgoSchema, and VSIBench. They also report scores on the "native" baseline, i.e. querying the model directly.

They find that SAVE seems to outperform some more compute/cost intensive methods of agentic-video-reasoning, and that the native-baseline is highly competitive to begin with. The authors show that agentic video reasoning is particularly beneficial for long videos and for questions whose relevant evidence is concentrated in a short segment.

To investigate the effect of evidence concentration, they construct a synthetic video question-answering benchmark comprising 384 videos ranging from 30 seconds to 10 minutes. The task requires agents to identify the sequence of color changes in stationary squares, with color-cycle durations varying across videos. SAVE performs best on longer videos with shorter color cycles, where the evidence needed to answer the question occupies only a small fraction of the video.

Critical strengths

The goal of systematic evaluation is good: standardizing backbones to compare methods is a reasonable way to better understand the value of existing systems, as well as their comparative strengths and weaknesses.

The synthetic benchmark is interesting for its systematic analysis of how the temporal concentration of evidence affects the need for agentic reasoning, extending beyond the well-studied role of video length.

Critical weaknesses
  • Too few backbones and benchmarks for an evaluation-focus paper. The value of each method might be better accentuated under different model sizes/families, which is not currently explored to support the claim that "agentic video reasoning should favor simple frameworks over increasingly elaborate pipelines". Moreover, ablations and analyses (such as on video length and temporal concentration) are only focused on single model backbones.

  • SAVE includes several interesting design choices, particularly its calibration of spatial resolution and frame sampling rate, and provides a useful baseline for evaluating more complex methods. However, its adaptive frame-selection strategy appears closely related to earlier work, such as AVUA (Jeoung et al., 2024), which is not discussed in the paper. An explicit comparison with this work would help clarify SAVE's distinct contributions. Reference: Jeoung et al. (2024). Adaptive Video Understanding Agent: Enhancing Efficiency with Dynamic Frame Sampling and Feedback-Driven Reasoning. Oral presentation at the NeurIPS 2024 Multimodal Algorithmic Reasoning Workshop.

  • The authors state that they reuse the baselines' original hyperparameters (line 256). However, hyperparameters tuned for one backbone may not transfer optimally to another. Tuning all methods on a common validation set would help ensure a fair comparison, particularly if SAVE's hyperparameters were selected through internal validation.

Questions for authors
  • Does model size matter for agentic workflows? Models such as Qwen3VL or SmolVLM2, which are available in multiple sizes, could be good candidates for exploring this question. Agentic frameworks might unlock potential in much smaller models.

  • While very interesting, the controlled comparison of video length and temporal concentration (Figure 1) is conducted only with Gemini 3 Flash. Is there a reason why it was not replicated with open-source models?

*Which model is used for the ablation study in Table 2? I assume it is Gemma 4 because efficiency is measured in tokens. It would be useful to make it more clear in the caption.

*How were SAVE's hyperparameters selected, and were the baselines given comparable tuning opportunities? If baseline hyperparameters were transferred unchanged from their original backbones, was their sensitivity evaluated on the backbones used here?

Reviewer_taRvOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper aims to study whether agentic video understanding framework is effective compared to using the backbone directly. The authors evaluate a diverse set of agentic video reasoning systems using two common backbones including Gemini 3 Flash and Gemma for controlled experiments. The authors find that most of the agentic frameworks they evaluate do not consistently outperform native video inference despite higher computational costs. Furthermore, the authors introduce a simple Active Exploration Baseline and an extension SAVE, which achieves better performance than most evaluated agentic baselines with lower inference costs. The paper also investigates when agentic video reasoning is preferable to native inference and find that the agentic system is more beneficial when the video length is long and temporal concentration is low.

Critical strengths
  1. The paper is very well written and easy to follow.
  2. The paper conducts controlled evaluation using shared backbones to compare different agentic frameworks and study whether complex agentic video reasoning frameworks are necessary.
  3. The proposed method SAVE is simple and effectively improves the base model performance with lower cost.
Critical weaknesses
  1. The agentic video understanding methods evaluated in this paper are mostly from 2024 and 2025, missing recent methods such as VideoSeek, LensWalk, and VideoARM (CVPR2026). These methods shows that the agentic system outperforms the base model, which could change the conclusion of the paper. The authors should evaluate the recent agentic systems and compare SAVE with them.
  2. Despite positioning the agentic video understanding systems against long video understanding scenarios, the paper do not evaluate on popular long video understanding benchmarks such as Video-MME, LongVideoBench and LVBench. These long video benchmarks will be helpful for both evaluating the existing agentic methods and the proposed SAVE.
  3. The main findings of the paper that agentic video reasoning helps most with long video and low temporal concentration seem largely intuitive and not very informative. Under a fixed token budget, uniform sampling naturally becomes less effective as video length increases, especially when fine-grained temporal information is needed. These are the scenarios that agentic video understanding frameworks are designed for.
Questions for authors
  1. Can the authors evaluate recent agentic video understanding methods and also compare SAVE with them?
  2. Can the authors provide evaluation on long-video benchmarks such as Video-MME, LongVideoBench and LVBench?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.