- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper introduces a framework for agentic video understanding, LARVA. Instead of relying on textual chain-of-thought and observation captions, LARVA uses a reasoner and an observer to communicate through compact continuous latent tokens, aiming to preserve useful visual information while reducing context length and inference overhead. Experiments across five video understanding benchmarks demonstrate improvements over textual CoT baselines based on Qwen2.5-VL-7B and Qwen3-VL-4B, along with reduced inference latency.
Critical strengths
- I find the idea behind this work interesting. It explores a middle ground between textual CoT and directly feeding raw video crops into the reasoning model, aiming to balance the preservation of fine-grained visual details with context length.
- The evaluation goes beyond final answer accuracy. The answerer-swapping experiments and human evaluation of evidence acquisition provide useful complementary evidence for assessing the effectiveness of the proposed approach.
Critical weaknesses
- I find the presentation somewhat difficult to follow. The proposed pipeline is relatively complex, involving separate reasoner and observer roles with different LoRA adapters. It also relies on the predefined Overview-Skim-Focus workflow from VideoSeek, which introduces relatively rigid hyperparameters. Training this pipeline also requires several carefully designed loss functions.
- I think the claim of motivation is somewhat biased toward caption-based video agents. Recent approaches can directly return visual content with dynamically allocated visual token budgets after selecting relevant temporal intervals through tool calls, rather than relying on an observer to generate textual captions (e.g., Video-o3). Although the paper briefly acknowledges this alternative, I believe the motivation would benefit from a more balanced discussion of such approaches.
- Some of the baselines seem relatively weak. In particular, I am not convinced that Qwen2.5-VL-7B-Instruct provides sufficiently strong reasoning capabilities, and this model version does not appear to have a dedicated thinking variant. This raises questions about whether the reported improvements would generalize to more recent thinking model baselines.
Video-o3: Native Interleaved Clue Seeking for Long Video Multi-Hop Reasoning
Questions for authors
- There appear to be some formatting issues in the manuscript. In particular, the line spacing around Lines 410–411 and 464–465 seems unusually tight.
5 comments
W2: I think the claim of motivation is somewhat biased toward caption-based video agents. Recent approaches can directly return visual content with dynamically allocated visual token budgets after selecting relevant temporal intervals through tool calls, rather than relying on an observer to generate textual captions (e.g., Video-o3). Although the paper briefly acknowledges this alternative, I believe the motivation would benefit from a more balanced discussion of such approaches.
W2. Motivation and comparison with direct visual input
Direct visual input is a meaningful alternative. We agree that agents such as Video-o3 are an important alternative to caption-based communication. We have revised the Introduction and related work to acknowledge temporal-region selection and dynamically allocated visual token budgets.
Our main tables already cover two direct-visual agent baselines. EVA selects temporal windows then directly inspects the acquired frames; VideoMind grounds and verifies temporal segments and answers from video input using role-specific LoRA adapters. Table 1 reports higher LARVA accuracy on all five benchmarks than both methods; on VideoHolmes/Video-MME/LVBench, the gains are 7.1/2.2/1.7 points over EVA and 3.3/4.2/4.2 over VideoMind. Table 3 also reports lower measured latency for LARVA compared to these methods.
We include a direct-visual CoT baseline alongside LARVA. The following Qwen2.5-VL-7B comparison uses the same tool harness and source trajectories. The first two rows compare caption-based and direct-visual CoT agents; the LARVA row shows the complete method:
| Reasoning | Observation interface | VideoHolmes | Video-MME | LVBench | Latency (s) |
|---|---|---|---|---|---|
| Textual CoT | Captions | 39.2 | 59.9 | 41.3 | 48.3 |
| Textual CoT | Raw visual tokens | 40.1 | 61.3 | 40.8 | 50.0 |
| Latent | Learned observation latents | 44.3 | 62.4 | 45.0 | 8.7 |
Direct visual input provides a complementary baseline. With textual CoT reasoning retained, direct visual input changes accuracy by +0.9/+1.4/−0.5 percentage points relative to captions. Full LARVA exceeds this direct-visual CoT baseline by 4.2/1.1/4.2 points.
W3: Some of the baselines seem relatively weak. In particular, I am not convinced that Qwen2.5-VL-7B-Instruct provides sufficiently strong reasoning capabilities, and this model version does not appear to have a dedicated thinking variant. This raises questions about whether the reported improvements would generalize to more recent thinking model baselines.
W3. Stronger baselines and generalization to thinking models
The expanded comparison covers a thinking backbone and a 27B backbone. Qwen3-VL-8B-Thinking uses native thinking with a 40,960-token budget and up to eight acquisition rounds in the off-the-shelf baseline. Its CoT-distilled student follows the paper’s CoT-baseline procedure. The table also includes Qwen3.8-27B with VideoSeek, VideoSeek-Image, and Codex, its CoT-distilled student, and its textual- versus learned-observation variants with latent reasoning fixed. All accuracies use the full benchmarks at 640 × 360 resolution. Appendix A.10 reports available latency measurements separately.
| Model and configuration | VideoHolmes | Video-MME | LVBench |
|---|---|---|---|
| 8B-Thinking: Off-the-shelf, VideoSeek | 36.5 | 57.3 | 40.8 |
| 8B-Thinking: Off-the-shelf, VideoSeek-Image | 37.7 | 57.1 | 43.4 |
| 8B-Thinking: CoT-distilled, VideoSeek | 44.2 | 59.0 | 42.7 |
| 8B-Thinking: Latent reasoning, textual observations | 46.4 | 62.9 | 44.1 |
| 8B-Thinking: LARVA, latent reasoning and learned observations | 50.7 | 65.8 | 47.4 |
| 27B: off-the-shelf, VideoSeek | 50.3 | 72.5 | 56.8 |
| 27B: off-the-shelf, VideoSeek-Image | 52.6 | 73.4 | 57.4 |
| 27B: off-the-shelf, Codex | 53.4 | 71.6 | 55.6 |
| 27B: CoT-distilled, VideoSeek | 48.1 | 71.4 | 52.9 |
| 27B: latent reasoning, textual observations | 52.3 | 72.5 | 57.5 |
| 27B: LARVA, learned observations | 55.6 | 74.7 | 59.5 |
Gains extend to both additional backbones. LARVA improves over the CoT-distilled student by 6.5/6.8/4.7 points on Qwen3-VL-8B-Thinking and 7.5/3.3/6.6 points on Qwen3.8-27B. With latent reasoning fixed, learned observations add 4.3/2.9/3.3 points and 3.3/2.2/2.0 points, respectively. On the 8B thinking backbone, this observation change also reduces latency from 42.2 to 9.3 seconds (4.54×). On 27B, full LARVA exceeds the off-the-shelf Codex baseline by 2.2/3.1/3.9 points and reduces latency from 121.2 to 28.7 seconds (4.22×). These results support the design on the evaluated backbones and settings.
The expanded comparison shows competitive accuracy and several frontier-model wins. On the matched 100 questions per benchmark, LARVA-27B achieves 57.0% on VideoHolmes, exceeding Astra/Qwen3.8-Max/DeepSeek/GLM at 55.0/52.0/54.0/49.0%. It exceeds GLM on all three benchmarks by 8.0/3.0/6.0 points and DeepSeek on Video-MME by 7.0 points, while remaining within one point of Qwen3.8-Max on Video-MME (79.0% versus 80.0%). Its reported 26.1 seconds per question is 2.86×–6.30× lower than these frontier baselines.
Q1: There appear to be some formatting issues in the manuscript. In particular, the line spacing around Lines 410–411 and 464–465 seems unusually tight.
Q1. Formatting around lines 410–411 and 464–465
Thank you for identifying the spacing issues. We have corrected the spacing around Section 4.4 and the paragraph near lines 464–465, and checked the compiled revision for consistent heading and paragraph spacing.
Thank you for highlighting direct-visual-input agents, baseline strength, and presentation clarity. We have expanded the discussion of direct visual interfaces, retained a direct-visual CoT baseline, and added experiments with a dedicated thinking backbone. These additions support the original claims about the joint latent loop, decision-supervised communication, and stronger video understanding with evidence-acquisition gains, together with the new fourth contribution on efficiency.
W1. Pipeline complexity, adapters, tools, and losses
The added design components improve the evaluated accuracy–latency trade-off. Simpler inference architectures are useful baselines, but fewer components do not by themselves imply higher accuracy or lower latency. With Qwen2.5-VL-7B, full LARVA exceeds a textual-CoT agent with raw visual observations by 4.2/1.1/4.2 points while reducing latency from 50.0 to 8.7 seconds (5.75×). On Qwen3.8-27B, LARVA exceeds the direct-visual VideoSeek-Image baseline by 3.0/1.3/2.1 points at 3.65× lower latency, and the Codex baseline by 2.2/3.1/3.9 points at 4.22× lower latency. These are complete-system comparisons. At inference, LARVA repeats three steps: select a tool call, encode the acquired frames into observation latents, and use them to select another call or answer. Reconstruction and CoT alignment are training-only objectives. Appendix A.3 specifies that a final answer ends execution without an observer call.
Two adapters share one backbone. The reasoner and observer share a backbone with separate LoRA adapters for their respective roles. The revised diagram and method description make the shared parameters, role-specific adapters, and training gradient paths explicit.
Tool choice is flexible within frame budgets. The tools constrain frame budgets without imposing a fixed Overview–Skim–Focus sequence. Overview samples 64 frames, and Skim/Focus sample 16 frames; the reasoner chooses tools, temporal intervals, and when to answer. Starting with Overview is not mandatory. We adopt VideoSeek’s tool interface for controlled comparisons and do not claim tool design as a contribution. The reasoner generates eight latents before each decision, including the final answer. Observation compression uses eight latents per frame for Overview/Skim and eight per Focus call: 512 latents for Overview, 128 for Skim, and eight for Focus, excluding timestamps.
Matched loss ablations justify the additional training objectives. All variants use the main Qwen2.5-VL-7B initialization, including the one-epoch CoT checkpoint, both components trainable, the same 20,000 trajectories, one epoch, learning rate 1e-4, and batch size 16. Removing reconstruction reduces accuracy from 44.3/62.4/45.0 to 35.5/45.0/34.7; removing alignment gives 38.8/57.9/38.2; action/answer supervision alone gives 31.3/43.6/32.5. These results support both auxiliary losses under the matched setup. Their role is to learn a better inference interface, and neither branch is executed at inference.