- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Summary. The paper performs "the broadest controlled evaluation to date" of recent agentic video-reasoning systems "across two strong contemporary backbones, with explicit token and monetary-cost accounting." It unifies agents under a POMDP-inspired abstraction, reimplements six on Gemini 3 Flash and Gemma 4, and introduces "a deliberately minimal Active Exploration Baseline" and SAVE, which "lets the backbone query temporal ranges and adaptively allocates temporal and visual resolution within them." Contributions. (1) The "most comprehensive controlled evaluation to date… with explicit computational accounting, showing that most evaluated frameworks fail to improve over native inference while often substantially increasing cost." (2) A minimal baseline that "remains competitive with substantially more elaborate agents," plus SAVE, which "outperforms all prior agentic approaches in most settings at substantially lower computational cost." (3) A characterization of when "agentic exploration is most useful, namely when uniform processing would represent answer-relevant evidence too sparsely." Validation. Evaluated "across three benchmarks and two backbones," SAVE reaches "the highest performance among evaluated prior agentic methods in five of six settings," hitting "66.4% with Gemini 3 Flash, comparable to 64.8% for native inference," "while reducing monetary inference cost by approximately 60%." Significance uses video-clustered permutation and McNemar tests with Holm correction; its "advantage over a token-matched native baseline grows with video duration," and a synthetic experiment shows adaptive inspection helps most "when the temporal pattern… occupies only a small fraction of the video."
Critical strengths
The paper's greatest strength is its "less is more" stance, which runs against the field's trend of ever more elaborate agents and is especially interesting as backbones grow stronger. The authors show that "when paired with strong multimodal backbones, most existing agentic frameworks fail to consistently outperform native video inference," and that "a deliberately minimal Active Exploration Baseline… remains competitive with substantially more elaborate agents." They conclude that "agentic video reasoning should favor simple frameworks over increasingly elaborate pipelines."
This claim is backed by solid empirics: a controlled evaluation under shared backbones with explicit cost accounting, rigorous significance testing (video-clustered permutation and McNemar tests with Holm correction), and a clear characterization of when agents help, namely "when uniform processing would represent answer-relevant evidence too sparsely."
Critical weaknesses
- Agentic gains unproven. Table 2 shows richer temporal context helps. But there is no fixed non-agentic multi-stage sampler to rule out that no planning is needed. SAOS does not close this gap. It "restricts SAVE to a single conditioned localization" and is itself agentic. The paper never shows "agentic machinery is actually necessary."
- Overclaim on two backbones. It claims "agents do not consistently improve video reasoning performance, irrespective of the evaluated framework." Yet it tests only Gemini 3 Flash and Gemma 4. The method is backbone-agnostic. Stronger closed models like GPT-5.6 or GPT-6 Astra and an independent open family like Qwen 3.8 27B could be run. None are tested or justified.
- Modest uncontrolled results. On Gemini "the difference between SAVE and direct Gemini 3 Flash is not statistically significant (pHolm = 0.390)." SAVE also trails native on VSIBench with 57.1 versus 59.2. The one clear win on Gemma Minerva uses a native baseline capped at "at most 60 [frames] per call" while SAVE accesses the video over many rounds.
- Toy synthetic missing analysis. The synthetic videos repeat an event throughout and remove localization difficulty yet motivate long-video claims. For an efficiency paper "we do not include end-to-end latency." There is no per-question-type or failure-case analysis
Questions for authors
- Non-agentic control. Can a fixed coarse-to-fine sampler with no LLM decisions also reach 66.4% on Minerva under the same budget? SAOS does not answer this. It "restricts SAVE to a single conditioned localization" and is still agentic. Can you add a pure heuristic multi-stage sampling baseline?
- Cross-family backbones. The framework is backbone-agnostic at the API level. Why evaluate only Gemini 3 Flash and Gemma 4? Does "most existing agentic frameworks fail to consistently improve over native video inference" still hold on GPT-5.6 or Qwen 3.8 27B?
- Controlled native comparison. Native Gemma is capped at "at most 60" frames per call while SAVE accesses the video over many rounds. If native gets the same total frame budget does the Gemma Minerva win remain? On Gemini "the difference between SAVE and direct Gemini 3 Flash is not statistically significant (pHolm = 0.390)." How does this support the overall claim?
- Efficiency and regime. You ask when agents are preferable yet "we do not include end-to-end latency." Can you report latency call counts and per-question-type plus failure-case analysis to mark when SAVE underperforms native?
- As frontier models rapidly improve, both closed (e.g., 6astra) and open (e.g., Qwen3.8-27B) models offer ever stronger native long-context video understanding. Is the video reasoning task this paper targets still a real problem? More importantly, relative to simply plugging such strong models into a general-purpose base harness (e.g., Codex), where does your method's value and irreplaceability lie? Please provide a direct comparison against a "base harness + frontier model" setup.