acceptodds
Withdrawn from ICLR 2027

Joint Latent Reasoning and Observation for Agentic Video Understanding

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Agentic video understanding requires models to acquire useful visual evidence through repeated tool interactions. Distilling this capability into smaller models requires transferring not only answer prediction, but also the teacher's evidence-acquisition strategy. Conventional distillation relies on textual chain-of-thought and verbalized observations, incurring autoregressive generation costs while potentially omitting visual details needed for subsequent decisions. We introduce Latent Agentic Reasoning for Video Analysis (LARVA), a framework that jointly learns latent reasoning and latent visual observations. A reasoner selects acquisition actions and produces answers, while an observer communicates selected visual evidence through compact continuous representations. We train LARVA on teacher trajectories generated from VideoMarathon and evaluate it on VideoHolmes, Video-MME, and LVBench. Our evaluations show significant advantages over directly distilled CoT models and recent agentic video-understanding methods. Trajectory analysis suggests that LARVA more closely follows the teacher's progression from broad inspection to targeted acquisition than a CoT-and-caption student. These observations motivate investigating joint latent distillation as a means of transferring effective evidence-acquisition strategies for video understanding.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multimodal, Video Understanding, Agent, Question-answering

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_vkFeOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces a framework for agentic video understanding, LARVA. Instead of relying on textual chain-of-thought and observation captions, LARVA uses a reasoner and an observer to communicate through compact continuous latent tokens, aiming to preserve useful visual information while reducing context length and inference overhead. Experiments across five video understanding benchmarks demonstrate improvements over textual CoT baselines based on Qwen2.5-VL-7B and Qwen3-VL-4B, along with reduced inference latency.

Critical strengths
  1. I find the idea behind this work interesting. It explores a middle ground between textual CoT and directly feeding raw video crops into the reasoning model, aiming to balance the preservation of fine-grained visual details with context length.
  2. The evaluation goes beyond final answer accuracy. The answerer-swapping experiments and human evaluation of evidence acquisition provide useful complementary evidence for assessing the effectiveness of the proposed approach.
Critical weaknesses
  1. I find the presentation somewhat difficult to follow. The proposed pipeline is relatively complex, involving separate reasoner and observer roles with different LoRA adapters. It also relies on the predefined Overview-Skim-Focus workflow from VideoSeek, which introduces relatively rigid hyperparameters. Training this pipeline also requires several carefully designed loss functions.
  2. I think the claim of motivation is somewhat biased toward caption-based video agents. Recent approaches can directly return visual content with dynamically allocated visual token budgets after selecting relevant temporal intervals through tool calls, rather than relying on an observer to generate textual captions (e.g., Video-o3). Although the paper briefly acknowledges this alternative, I believe the motivation would benefit from a more balanced discussion of such approaches.
  3. Some of the baselines seem relatively weak. In particular, I am not convinced that Qwen2.5-VL-7B-Instruct provides sufficiently strong reasoning capabilities, and this model version does not appear to have a dedicated thinking variant. This raises questions about whether the reported improvements would generalize to more recent thinking model baselines.

Video-o3: Native Interleaved Clue Seeking for Long Video Multi-Hop Reasoning

Questions for authors
  1. There appear to be some formatting issues in the manuscript. In particular, the line spacing around Lines 410–411 and 464–465 seems unusually tight.
5 comments

Thank you for highlighting direct-visual-input agents, baseline strength, and presentation clarity. We have expanded the discussion of direct visual interfaces, retained a direct-visual CoT baseline, and added experiments with a dedicated thinking backbone. These additions support the original claims about the joint latent loop, decision-supervised communication, and stronger video understanding with evidence-acquisition gains, together with the new fourth contribution on efficiency.

W1: I find the presentation somewhat difficult to follow. The proposed pipeline is relatively complex, involving separate reasoner and observer roles with different LoRA adapters. It also relies on the predefined Overview-Skim-Focus workflow from VideoSeek, which introduces relatively rigid hyperparameters. Training this pipeline also requires several carefully designed loss functions.

W1. Pipeline complexity, adapters, tools, and losses

The added design components improve the evaluated accuracy–latency trade-off. Simpler inference architectures are useful baselines, but fewer components do not by themselves imply higher accuracy or lower latency. With Qwen2.5-VL-7B, full LARVA exceeds a textual-CoT agent with raw visual observations by 4.2/1.1/4.2 points while reducing latency from 50.0 to 8.7 seconds (5.75×). On Qwen3.8-27B, LARVA exceeds the direct-visual VideoSeek-Image baseline by 3.0/1.3/2.1 points at 3.65× lower latency, and the Codex baseline by 2.2/3.1/3.9 points at 4.22× lower latency. These are complete-system comparisons. At inference, LARVA repeats three steps: select a tool call, encode the acquired frames into observation latents, and use them to select another call or answer. Reconstruction and CoT alignment are training-only objectives. Appendix A.3 specifies that a final answer ends execution without an observer call.

Two adapters share one backbone. The reasoner and observer share a backbone with separate LoRA adapters for their respective roles. The revised diagram and method description make the shared parameters, role-specific adapters, and training gradient paths explicit.

Tool choice is flexible within frame budgets. The tools constrain frame budgets without imposing a fixed Overview–Skim–Focus sequence. Overview samples 64 frames, and Skim/Focus sample 16 frames; the reasoner chooses tools, temporal intervals, and when to answer. Starting with Overview is not mandatory. We adopt VideoSeek’s tool interface for controlled comparisons and do not claim tool design as a contribution. The reasoner generates eight latents before each decision, including the final answer. Observation compression uses eight latents per frame for Overview/Skim and eight per Focus call: 512 latents for Overview, 128 for Skim, and eight for Focus, excluding timestamps.

Matched loss ablations justify the additional training objectives. All variants use the main Qwen2.5-VL-7B initialization, including the one-epoch CoT checkpoint, both components trainable, the same 20,000 trajectories, one epoch, learning rate 1e-4, and batch size 16. Removing reconstruction reduces accuracy from 44.3/62.4/45.0 to 35.5/45.0/34.7; removing alignment gives 38.8/57.9/38.2; action/answer supervision alone gives 31.3/43.6/32.5. These results support both auxiliary losses under the matched setup. Their role is to learn a better inference interface, and neither branch is executed at inference.

W2: I think the claim of motivation is somewhat biased toward caption-based video agents. Recent approaches can directly return visual content with dynamically allocated visual token budgets after selecting relevant temporal intervals through tool calls, rather than relying on an observer to generate textual captions (e.g., Video-o3). Although the paper briefly acknowledges this alternative, I believe the motivation would benefit from a more balanced discussion of such approaches.

W2. Motivation and comparison with direct visual input

Direct visual input is a meaningful alternative. We agree that agents such as Video-o3 are an important alternative to caption-based communication. We have revised the Introduction and related work to acknowledge temporal-region selection and dynamically allocated visual token budgets.

Our main tables already cover two direct-visual agent baselines. EVA selects temporal windows then directly inspects the acquired frames; VideoMind grounds and verifies temporal segments and answers from video input using role-specific LoRA adapters. Table 1 reports higher LARVA accuracy on all five benchmarks than both methods; on VideoHolmes/Video-MME/LVBench, the gains are 7.1/2.2/1.7 points over EVA and 3.3/4.2/4.2 over VideoMind. Table 3 also reports lower measured latency for LARVA compared to these methods.

We include a direct-visual CoT baseline alongside LARVA. The following Qwen2.5-VL-7B comparison uses the same tool harness and source trajectories. The first two rows compare caption-based and direct-visual CoT agents; the LARVA row shows the complete method:

ReasoningObservation interfaceVideoHolmesVideo-MMELVBenchLatency (s)
Textual CoTCaptions39.259.941.348.3
Textual CoTRaw visual tokens40.161.340.850.0
LatentLearned observation latents44.362.445.08.7

Direct visual input provides a complementary baseline. With textual CoT reasoning retained, direct visual input changes accuracy by +0.9/+1.4/−0.5 percentage points relative to captions. Full LARVA exceeds this direct-visual CoT baseline by 4.2/1.1/4.2 points.

W3: Some of the baselines seem relatively weak. In particular, I am not convinced that Qwen2.5-VL-7B-Instruct provides sufficiently strong reasoning capabilities, and this model version does not appear to have a dedicated thinking variant. This raises questions about whether the reported improvements would generalize to more recent thinking model baselines.

W3. Stronger baselines and generalization to thinking models

The expanded comparison covers a thinking backbone and a 27B backbone. Qwen3-VL-8B-Thinking uses native thinking with a 40,960-token budget and up to eight acquisition rounds in the off-the-shelf baseline. Its CoT-distilled student follows the paper’s CoT-baseline procedure. The table also includes Qwen3.8-27B with VideoSeek, VideoSeek-Image, and Codex, its CoT-distilled student, and its textual- versus learned-observation variants with latent reasoning fixed. All accuracies use the full benchmarks at 640 × 360 resolution. Appendix A.10 reports available latency measurements separately.

Model and configurationVideoHolmesVideo-MMELVBench
8B-Thinking: Off-the-shelf, VideoSeek36.557.340.8
8B-Thinking: Off-the-shelf, VideoSeek-Image37.757.143.4
8B-Thinking: CoT-distilled, VideoSeek44.259.042.7
8B-Thinking: Latent reasoning, textual observations46.462.944.1
8B-Thinking: LARVA, latent reasoning and learned observations50.765.847.4
27B: off-the-shelf, VideoSeek50.372.556.8
27B: off-the-shelf, VideoSeek-Image52.673.457.4
27B: off-the-shelf, Codex53.471.655.6
27B: CoT-distilled, VideoSeek48.171.452.9
27B: latent reasoning, textual observations52.372.557.5
27B: LARVA, learned observations55.674.759.5

Gains extend to both additional backbones. LARVA improves over the CoT-distilled student by 6.5/6.8/4.7 points on Qwen3-VL-8B-Thinking and 7.5/3.3/6.6 points on Qwen3.8-27B. With latent reasoning fixed, learned observations add 4.3/2.9/3.3 points and 3.3/2.2/2.0 points, respectively. On the 8B thinking backbone, this observation change also reduces latency from 42.2 to 9.3 seconds (4.54×). On 27B, full LARVA exceeds the off-the-shelf Codex baseline by 2.2/3.1/3.9 points and reduces latency from 121.2 to 28.7 seconds (4.22×). These results support the design on the evaluated backbones and settings.

The expanded comparison shows competitive accuracy and several frontier-model wins. On the matched 100 questions per benchmark, LARVA-27B achieves 57.0% on VideoHolmes, exceeding Astra/Qwen3.8-Max/DeepSeek/GLM at 55.0/52.0/54.0/49.0%. It exceeds GLM on all three benchmarks by 8.0/3.0/6.0 points and DeepSeek on Video-MME by 7.0 points, while remaining within one point of Qwen3.8-Max on Video-MME (79.0% versus 80.0%). Its reported 26.1 seconds per question is 2.86×–6.30× lower than these frontier baselines.

Q1: There appear to be some formatting issues in the manuscript. In particular, the line spacing around Lines 410–411 and 464–465 seems unusually tight.

Q1. Formatting around lines 410–411 and 464–465

Thank you for identifying the spacing issues. We have corrected the spacing around Section 4.4 and the paragraph near lines 464–465, and checked the compiled revision for consistent heading and paragraph spacing.

Reviewer_BgQnOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Summary LARVA distills agentic long-video understanding into small models. Unlike video agents that pass reasoning and observations as text, its reasoner and observer share a backbone with separate LoRA adapters and communicate via continuous latent vectors, with answer/action gradients flowing back to shape the observation encoding; it trains with action/answer, observation-reconstruction, and CoT-alignment losses, replaying teacher trajectories in training and acting autonomously at inference. Contributions A dual-latent interface for reasoning and observation where downstream gradients shape the observation encoding; a three-loss distillation framework transferring multi-round video-agent ability to 4B/7B models; and results on five long-video benchmarks beating same-teacher CoT baselines with fewer rounds, frames, and latency, supported by answerer-swap and human evaluation of evidence quality. Validation Evaluated on LVBench, LongVideoBench, Video-MME (Long), MLVU, and VideoHolmes with 4B and 7B backbones against CoT and Direct baselines; interface ablations separate latent reasoning from latent observation; an inference-cost table reports rounds, tokens, frames, and latency; answerer-swap and a 100-question human study assess evidence quality. Thorough overall, but the three losses are not ablated individually, the inference token counts are inconsistent with the setup, and the non-jointly-trained latent/latent baseline may be understated.

Critical strengths
  1. Insightful interface design Placing observations in a continuous latent space and letting answer/action gradients flow back to shape the observation encoding jointly trains how evidence is encoded and how it is used. This targets the core limitation of textual observations and fills a gap left open by prior work in the video-agent loop.
  2. Evidence beyond raw accuracy The answerer-swap (gathered evidence fed to a fixed 27B answerer still yields gains) and a 100-question human study of how often extra acquisitions add new evidence support the harder claim of higher evidence quality, not just end-to-end accuracy.
  3. Verifiable efficiency gains While surpassing same-teacher CoT baselines on five long-video benchmarks, it markedly cuts reasoning rounds, frames, and latency, reported item by item, which is practically meaningful for deploying small models.
Critical weaknesses
  1. Weak attribution of the core mechanism Interface ablations show the main gains come from replacing reasoning with latent (Latent/Textual is already close to the full model), while the headline dual-latent-plus-joint-training design adds only a fraction of a point on its own. The paper's primary novelty thus lacks clean evidence, and it is hard to tell whether latent observations are essential.
  2. Loss and baseline choices weaken the causal claims The three losses (action/answer, observation reconstruction, CoT alignment) are combined at λ=1 with no individual ablation, so their separate contributions, especially whether CoT alignment and reconstruction are needed, are unclear. The non-jointly-trained latent/latent baseline freezes a text-trained reasoner before adapting the observer, an interface mismatch that likely understates it and overstates the benefit of joint training.
  3. Efficiency accounting and reproducibility concerns The reasoning token count in the cost table is inconsistent with the stated setup (per-round tokens × rounds), leaving end-to-end cost unclear; combined with the limited teacher-trajectory coverage after incorrect acquisitions (a stated limitation), this reduces confidence in the reported efficiency and evidence-quality claims.
Questions for authors
  1. In the interface ablation, Latent/Textual is already close to the full model, so can you report the net gain of adding latent observations alone (fixed latent reasoning) over Latent/Textual, to show latent observations are necessary?
  2. Can each loss be ablated individually (removing observation reconstruction and removing CoT alignment separately), and what motivates the λ=1 choice?
  3. Is the non-jointly-trained latent/latent baseline understated because of the interface mismatch from freezing a text-trained reasoner before adapting the observer? Could you add a jointly-trained control that only removes the gradient flow-back?
  4. How is the reasoning token count in the cost table measured? It appears inconsistent with per-round tokens × rounds; does the end-to-end cost include the observer encoding and all teacher-independent overhead?
  5. As frontier models rapidly improve, both closed (e.g., 6astra) and open (e.g., Qwen3.8-27B) models offer ever stronger native long-context video understanding. Is the video reasoning task this paper targets still a real problem? More importantly, relative to simply plugging such strong models into a general-purpose base harness (e.g., Codex), where does your method's value and irreplaceability lie? Please provide a direct comparison against a "base harness + frontier model" setup.
5 comments

Thank you for the detailed assessment and for recognizing the continuous interface, evidence-acquisition analysis, and efficiency results. Learned observations improve evidence gathering: under the same additional-frame budget, LARVA acquires new answer-relevant evidence on 35% of questions versus 15% with textual observations. We complement this result with accuracy gains on 4B, 8B thinking, and 27B backbones, a matched downstream-gradient control, corrected efficiency accounting, and direct comparisons with stronger models and general-purpose harnesses.

W1: Weak attribution of the core mechanism Interface ablations show the main gains come from replacing reasoning with latent (Latent/Textual is already close to the full model), while the headline dual-latent-plus-joint-training design adds only a fraction of a point on its own. The paper's primary novelty thus lacks clean evidence, and it is hard to tell whether latent observations are essential.

Q1: In the interface ablation, Latent/Textual is already close to the full model, so can you report the net gain of adding latent observations alone (fixed latent reasoning) over Latent/Textual, to show latent observations are necessary?

W1 / Q1. Performance, efficiency, and evidence-acquisition contributions of latent observations

Learned observations improve the 4B, 8B thinking, and 27B backbones. Keeping the reasoning interface latent isolates the incremental observation comparison within each backbone. Accuracy improves by 4.7/1.5/1.1 percentage points on Qwen3-VL-4B, by 4.3/2.9/3.3 points on Qwen3-VL-8B-Thinking, and by 3.3/2.2/2.0 points on Qwen3.8-27B, in VideoHolmes/Video-MME/LVBench order (Table 10). The 27B scores increase from 52.3/72.5/57.5 to 55.6/74.7/59.5. These results extend the observation benefit to the evaluated thinking and larger backbones.

BackboneObservation interfaceVideoHolmesVideo-MMELVBench
Qwen3-VL-8B-ThinkingTextual46.462.944.1
Qwen3-VL-8B-ThinkingLearned latent50.765.847.4
Qwen3-VL-8B-ThinkingGain+4.3+2.9+3.3
Qwen3-VL-4BTextual43.560.640.2
Qwen3-VL-4BLearned latent48.262.141.3
Qwen3-VL-4BGain+4.7+1.5+1.1
Qwen3.8-27BTextual52.372.557.5
Qwen3.8-27BLearned latent55.674.759.5
Qwen3.8-27BGain+3.3+2.2+2.0

Learned observations improve efficiency as well as accuracy. With latent reasoning retained, Qwen3-VL-8B-Thinking takes 42.2 seconds with textual observations and 9.3 seconds with learned observations, a 4.54× speedup alongside the accuracy gains above. These full-benchmark measurements use one H200, batch size 1, BF16, and 640 × 360 resolution under the same end-to-end timing protocol (Table 12). They include observer and reasoner computation and any differences in acquisition behavior. This is direct evidence for our efficiency contribution.

A matched gradient control supports the training mechanism. Keeping both adapters trainable and blocking only action/answer gradients into the observer reduces accuracy by 2.6/2.1/3.4 points. This control, detailed in W2 / Q3 below and revised Table 8, supports the contribution of downstream supervision separately from the interface comparisons above.

LARVA gathers more useful evidence under the same frame budget. In the 100-question human study, full LARVA obtains new answer-relevant evidence on 35% of questions, compared with 15% for latent reasoning with textual observations: a 20-percentage-point gain after the same 64-frame Overview and one additional 16-frame acquisition. This directly evaluates where the agent looks next, independently of its final answer. Textual reasoning with latent observations yields 11%. Two independent judges assess anonymized acquisitions, disagreements are adjudicated, and uncertain cases remain in the denominator (Table 5). Appendix A.11 adds acquisition successes in entity–attribute binding and event ordering, as well as an occurrence-counting failure. The conclusion is better evidence gathering for the tested acquisition step, not guaranteed coverage or successful recovery over an entire trajectory.

W2: Loss and baseline choices weaken the causal claims The three losses (action/answer, observation reconstruction, CoT alignment) are combined at λ=1 with no individual ablation, so their separate contributions, especially whether CoT alignment and reconstruction are needed, are unclear. The non-jointly-trained latent/latent baseline freezes a text-trained reasoner before adapting the observer, an interface mismatch that likely understates it and overstates the benefit of joint training.

Q2: Can each loss be ablated individually (removing observation reconstruction and removing CoT alignment separately), and what motivates the λ=1 choice?

W2 / Q2. Individual loss ablations and λ = 1

Loss ablations use matched training conditions. We add matched loss ablations on Qwen2.5-VL-7B. All variants use the main-model initialization, including the one-epoch CoT checkpoint, with both components trainable, the same 20,000 trajectories, one epoch, learning rate 1e-4, and batch size 16.

Training objectiveVideoHolmesVideo-MMELVBench
Full objective44.362.445.0
Without observation reconstruction35.545.034.7
Without CoT alignment38.857.938.2
Action/answer supervision alone31.343.632.5

Both auxiliary losses contribute under the matched setup. Each single removal retains the other two losses, and both removals substantially reduce accuracy. The action/answer-only variant performs worse still.

Loss-weight sensitivity is evaluated on held-out videos. For loss-weight sensitivity, we report QA accuracy (%) on 1,000 randomly sampled held-out VideoMarathon questions, with videos disjoint from training. Each column varies only the named coefficient; the other two remain 1. The revised Appendix A.9 specifies that this sensitivity sweep is limited to the tested coefficient range and does not establish globally optimal weights.

Coefficient valueVary λobsλ_\text{obs}Vary λalignλ_\text{align}
0.5066.270.7
0.7571.573.5
1.0074.274.2
1.2573.872.0

Unit loss weights are a strong default. We use 1 as a common default because increasing the coefficients above 1 yields no additional benefit: reconstruction decreases by 0.4 points, whereas alignment decreases by 2.2 points.

W2: Loss and baseline choices weaken the causal claims The three losses (action/answer, observation reconstruction, CoT alignment) are combined at λ=1 with no individual ablation, so their separate contributions, especially whether CoT alignment and reconstruction are needed, are unclear. The non-jointly-trained latent/latent baseline freezes a text-trained reasoner before adapting the observer, an interface mismatch that likely understates it and overstates the benefit of joint training.

Q3: Is the non-jointly-trained latent/latent baseline understated because of the interface mismatch from freezing a text-trained reasoner before adapting the observer? Could you add a jointly-trained control that only removes the gradient flow-back?

W2 / Q3. Frozen-reasoner mismatch and gradient flow

We agree that the original non-joint baseline is confounded by receiver adaptation: it fits a latent observer to a frozen reasoner previously trained with textual observations. Its 5.3–7.2-point gap should not be interpreted as a clean estimate of downstream-gradient benefits.

A matched control isolates downstream gradients. We train a control with the same initialization, latent interfaces, data, and optimization schedule as LARVA. Both adapters remain trainable. We block only the action/answer gradient into the observer while retaining its reconstruction and alignment supervision; the reasoner continues to learn from action/answer supervision.

Gradients reaching the observerVideoHolmesVideo-MMELVBench
Reconstruction + alignment + action/answer44.362.445.0
Reconstruction + alignment; action/answer blocked41.760.341.6
Reconstruction only; alignment and action/answer blocked41.159.939.7

Downstream feedback provides a measurable gain. The matched action/answer-blocked control isolates a 2.6/2.1/3.4-point benefit from downstream action/answer feedback into observations. The final row additionally blocks alignment gradients into the observer; “reconstruction only” describes observer supervision.

W3: Efficiency accounting and reproducibility concerns The reasoning token count in the cost table is inconsistent with the stated setup (per-round tokens × rounds), leaving end-to-end cost unclear; combined with the limited teacher-trajectory coverage after incorrect acquisitions (a stated limitation), this reduces confidence in the reported efficiency and evidence-quality claims.

Q4: How is the reasoning token count in the cost table measured? It appears inconsistent with per-round tokens × rounds; does the end-to-end cost include the observer encoding and all teacher-independent overhead?

W3 / Q4. Corrected token accounting, latency, and trajectory coverage

We correct the efficiency checkpoint mismatch. Thank you for identifying the inconsistency. All accuracy results use the checkpoint with eight reasoning latents per decision, but the original efficiency measurements inadvertently used an obsolete checkpoint with four reasoning latents per decision. The observer configuration was unchanged: eight observation latents per frame for Overview/Skim and eight per Focus call. We replace the efficiency measurements with those from the same eight-reasoning-latent checkpoint used for accuracy:

ModelTurnsReasoning latentsObservation latentsFramesLatency (s)
LARVA, eight reasoning latents per decision3.427.2579.677.38.7

Corrected accounting confirms fewer turns and frames. A turn counts each agent output, including the final answer, for both CoT and LARVA. Thus, 3.4 mean turns correspond to 3.4 × 8 = 27.2 reasoning latents. Total turns decrease from 4.8 to 3.4 (29.2%); excluding the terminal answer, acquisition calls decrease from 3.8 to 2.4 (36.8%). Processed frames decrease from 112.5 to 77.3 (31.3%). Observation counts include only latents, excluding timestamp tokens. Overview is not mandatory at the start. We have corrected the affected method descriptions, table entries, and derived speedups. Relative to the 48.3 s CoT and 39.0 s Direct baselines, the corrected speedups are 5.55× and 4.48×. Table 3 reports 579.6 observation latents on average and compares complete configurations.

Latency includes end-to-end student inference. Local timing uses one H200 GPU, batch size 1, and BF16. The corrected 7B values above are pooled averages over all questions from the three diagnostic benchmarks. Timing begins with the question and video in memory and ends with the final answer. It includes frame sampling, preprocessing, visual encoding, observer generation, reasoner computation, and adapter switching where applicable; only video-file loading into memory is excluded. Teacher inference and the training-only auxiliary branches are not used during student inference. The manuscript now specifies 640 × 360 resolution for this pooled measurement, as well as the full timing boundary and included preprocessing stages.

Recovery after student mistakes remains a limitation. Teacher replay can underrepresent histories following student mistakes. Our evaluations use autonomous student actions and therefore include their consequences, but recovery after an erroneous acquisition has not been separately measured. We retain this as a limitation rather than claiming autonomous evaluation resolves it.

Reviewer_fjebOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper aims to address the limitations of existing agentic video understanding methods that rely on textual intermediate reasoning and observations. Motivated by the information loss and computational overhead introduced by textual communication, the paper argues for jointly learning latent reasoning and visual observations to improve both evidence acquisition and video understanding.

The authors proposes LARVA, a framework that jointly trains a latent reasoner and a latent observer. The reasoner determines which video segments to inspect, while the observer encodes the acquired visual evidence into compact continuous representations instead of textual descriptions. The framework is trained through three objectives: action and answer supervision, observation reconstruction, and CoT-distribution alignment, allowing downstream decisions to guide the learning of latent representations.

The proposed framework is built on Qwen2.5-VL-7B and Qwen3-VL-4B. Evaluations are conducted on VideoHolmes, Video-MME, LVBench, LongVideoBench, and MLVU. Experimental results demonstrate improvements in both accuracy and inference efficiency compared with textual CoT baselines.

Critical strengths
  • Overall, I appreciate the motivation of this paper and its understanding of the limitations of text-based communication in existing agentic video understanding frameworks. The idea of jointly learning latent reasoning and visual observations is interesting and well-motivated.
  • The proposed joint optimization of the reasoner and observer seems to be a promising approach.
  • The paper is well-written and easy to follow.
Critical weaknesses

My main concerns lie in:

  • The contribution of each training objective is not sufficiently investigated. The proposed framework combines action supervision, observation reconstruction, and CoT-distribution alignment. However, the paper lacks ablation studies on each component.
  • Although Fig 1 provides an illustrative example showing how the proposed LARVA avoid identity confusion and improves evidence acquisition compared with the textual CoT baseline, the qualitative analysis remains limited. It would be helpful to include more diverse success and failure cases to better understand when latent observations are beneficial and when they may fail, particularly for fine-grained visual details and temporal reasoning.
Questions for authors

Please see the weaknesses. It would be helpful to clarify how much each component contributes to the overall performance improvements.

2 comments

Thank you for recognizing the motivation, joint reasoner–observer design, and clarity of our paper. We have added matched objective and gradient controls and expanded the qualitative analysis. These additions support the paper’s contributions: the joint latent observation–reasoning–action loop, learning visual communication from downstream decisions, and stronger video understanding with evidence-acquisition gains.

W1: The contribution of each training objective is not sufficiently investigated. The proposed framework combines action supervision, observation reconstruction, and CoT-distribution alignment. However, the paper lacks ablation studies on each component.

W1. Contributions of the training objectives

Matched loss ablations isolate each objective. We add loss ablations using Qwen2.5-VL-7B, with the same initialization as the main model, including the one-epoch CoT checkpoint. Both the reasoner and the observer remain trainable. All variants use the same 20,000 trajectories, one training epoch, a learning rate of 1e-4, a batch size of 16, and the same evaluation protocol.

Training objectiveVideoHolmesVideo-MMELVBench
Full objective44.362.445.0
Without observation reconstruction35.545.034.7
Without CoT alignment38.857.938.2
Action/answer supervision alone31.343.632.5

Both auxiliary objectives improve accuracy. Removing reconstruction retains alignment and action/answer supervision; removing alignment retains reconstruction and action/answer supervision. Relative to the full objective, removing reconstruction reduces accuracy by 8.8/17.4/10.3 percentage points, and removing alignment reduces it by 5.5/4.5/6.8 percentage points. Retaining only action/answer supervision performs worse still. These results establish that both auxiliary objectives contribute under the matched training setup.

Downstream feedback benefits the observer. We also examine the specific contribution of downstream action/answer supervision to the observer. Keeping both adapters trainable, we block the action/answer gradient at the observation interface while retaining reconstruction and alignment supervision for the observer. The reasoner continues to receive action/answer supervision. Accuracy becomes 41.7/60.3/41.6, compared with 44.3/62.4/45.0 for full LARVA: downstream action/answer feedback improves accuracy by 2.6/2.1/3.4 points. This control tests the gradient path into the observer, rather than removing action/answer supervision from the entire model.

W2: Although Fig 1 provides an illustrative example showing how the proposed LARVA avoid identity confusion and improves evidence acquisition compared with the textual CoT baseline, the qualitative analysis remains limited. It would be helpful to include more diverse success and failure cases to better understand when latent observations are beneficial and when they may fail, particularly for fine-grained visual details and temporal reasoning.

W2. More diverse qualitative analysis

New examples cover successes and failures in observation and acquisition. Appendix A.11 (Figures 4–8) adds five cases: entity–attribute binding, fine-detail recognition, temporal counting, event ordering, and a failure in occurrence counting. The binding and event-ordering examples show how different acquisition histories affect the evidence available to answer; the fine-detail and temporal-counting cases illustrate differences in using observations. The failure case shows that additional acquisition does not guarantee a correct count. The panels include recorded calls and answers alongside illustrative frames, extending Figure 1 and the evidence-acquisition analysis.

Dear Reviewer fjeb, thank you again for your helpful feedback. Our rebuttal includes matched loss ablations, a gradient-flow control, and additional qualitative success and failure cases addressing your concerns. We would appreciate your thoughts on whether these additions resolve your questions, and would be happy to clarify any remaining points. Thank you for your time.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.