acceptodds
Under review as a conference paper at ICLR 2027

ViLaMAS: Latent State Handoffs for Multi-Agent Long-Video Understanding

Abstract

Multi-agent systems (MAS) for long-video question answering face a key bottleneck: progressive accuracy degradation across successive agent handoffs. Textual handoffs can omit visual evidence and intermediate reasoning, which may cause additional collaboration stages to undermine earlier understanding and limit the effective use of more elaborate MAS. We introduce ViLaMAS, a training-free framework for reducing this degradation within existing MAS workflows. Agents sharing a compatible frozen backbone transfer their complete accumulated multimodal KV caches and continuation metadata, allowing subsequent agents to continue autoregressive latent reasoning. Evidence-guided revisiting augments the inherited state with newly sampled frames and selected original frames, followed by latent updates that integrate the additional evidence. We also introduce Self-Verification Inference (SVI) as a baseline for long-video question answering. With Qwen3-VL-32B, SVI exceeds the reported accuracy of GPT-4o on EgoSchema and LongVideoBench. SVI with two textual agent handoffs reduces accuracy from 66.0% to 47.5% with Qwen3-VL-8B on LongVideoBench. ViLaMAS raises accuracy to 61.8% within the same collaboration structure. Across six MAS architectures, ViLaMAS reduces this gap by more than 60% on average compared with textual handoffs. These results support preserving and extending multimodal context to retain video understanding accuracy across compatible MAS.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multi-agent systems, long video understanding

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_h42SOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the problem of progressive accuracy degradation across agent handoffs in multi-agent long-video question answering (MAS for long-video QA). The core hypothesis is that textual handoffs can omit visual evidence and intermediate reasoning, thereby undermining the video understanding established by earlier agents. The authors propose ViLaMAS, a training-free framework that preserves and extends multimodal context for downstream agents through KV cache transfer, autoregressive latent continuation, and evidence-guided revisiting. They also introduce the SVI baseline and validate the method on multiple video understanding benchmarks, claiming it significantly narrows the gap relative to textual handoffs. However, the method is effective only in the restricted setting of a shared frozen backbone, and lacks theoretical guarantees as well as validation across heterogeneous models.

Critical strengths
  1. The authors quantitatively verified, through controlled experiments, that text-based handoffs in multi-agent systems lead to progressive accuracy degradation, providing a valuable empirical observation for the field of multi-agent video understanding.

  2. Through component-wise cumulative ablations starting from SVI, the paper reveals the marginal contribution of each module, which to some extent answers the question of "where the improvement comes from."

Critical weaknesses
  1. The core components are combinations of existing techniques, more akin to engineering integration than fundamental innovation. KV cache reuse/transfer and the mapping of hidden states back into the input embedding space have already been demonstrated in prior work (e.g., LatentMAS and related latent-communication methods), and similar evidence-revisiting mechanisms also appear in VideoMiner. The paper fails to clearly articulate the operational differences between ViLaMAS and these existing works.

  2. The proposed method can evidently only be applied to multi-agent scenarios with shared architecture and weights, imposing a significant limitation on generalization.

  3. The main experimental results (Table 1) show that the multi-agent variants (ViLaMAS) consistently underperform SVI. The paper does not clearly explain the rationale for adopting a multi-agent system at all, given that it fails to outperform the single-system SVI baseline. The motivation and added value of the multi-agent design remain unsubstantiated.

Questions for authors
  1. The paper needs a formal statement and proof of the equivalence between latent continuation after KV transfer and uninterrupted reasoning, and, where possible, an analysis of how error propagates (or is bounded) as handoff depth H grows.

  2. How do we know the accuracy gain comes from preserving context itself, rather than from extra latent update steps, more revisited frames, or more total compute?

  3. Do SVI and ViLaMAS differ significantly in inference FLOPs, latency, and peak GPU memory? This matters for pinning down the cost-benefit trade-off.

  4. In Equation 2, vi=fθ(VS0,q,C,oi;pver)v_i = fθ(V_{S0}, q, C, o_i; p_{ver}), the definition and form of viv_i should be given at the point where it first appears.

  5. The relation between ViLaMAS-Text and ViLaMAS should also be explained at first mention (line 187).

6 comments
AuthorsOpenReview ↗

Response to Reviewer h42S [1/4]

We sincerely thank you for your time and valuable questions.

W #1 The core components are combinations of existing techniques, more akin to engineering integration than fundamental innovation. KV cache reuse/transfer and the mapping of hidden states back into the input embedding space have already been demonstrated in prior work (e.g., LatentMAS and related latent-communication methods), and similar evidence-revisiting mechanisms also appear in VideoMiner. The paper fails to clearly articulate the operational differences between ViLaMAS and these existing works.

Response to W1:

Thank you for your feedback. We would like to clarify our methodological novelty and comparison with existing works:

W1-1 Innovation: Our work's novelty lies in integrating visual evidence directly into inter-agent communication and reasoning through latent-state updates, without reducing visual information to textual representations. The results of Table R1 support our innovation in integrating visual information into latent reasoning, bringing new capabilities. Meanwhile, Table R7 also demonstrates that we ensure inference efficiency while improving accuracy.

Table R1: Component-wise Accuracy Ablation on LongVideoBench (Qwen3-VL-32B).

MetricBaseline (Text)LU OnlyFull-KV OnlyViLaMAS (Ours)
Acc. (%) ↑51.452.155.964.1 (+ 12.7)

W1-2 Comparison with LatentMAS: LatentMAS focuses on KV-based latent-state communication, but ViLaMAS focuses on addressing multimodal information loss across agent handoffs through effective visual evidence selection and its integration into subsequent latent updates. Table R2 provides detailed comparison.

Table R2: Comparison on LongVideoBench with Qwen3-VL-8B/32B. LatentMAS (SVI-based) denotes our controlled best-effort multimodal adaptation initialized from the SVI fusion state. 47.6/52.2 are new SVI-based adaptation with LatentMAS's framework.

MethodFull-KVState-conditioned Video LocalizationMultiple MAS Architecture Adaptation (>7)Latent Updates with New Visual Evidence8B Acc. (%)32B Acc. (%)
LatentMAS (SVI-based)✓✗✗✗47.652.2
ViLaMAS (Ours)✓✓✓✓61.864.1

We also conduct experiments to compare our method with conventional latent reasoning approaches across existing MAS workflows. As shown in Table R3, LU alone yields only marginal improvements across different MAS architectures. In contrast, selectively incorporating relevant visual evidence into latent updates not only enhances multimodal information preservation but also unlocks the potential of LU for mitigating handoff-induced information loss.

Table R3: LU and EGL ablations across representative MAS workflows on LongVideoBench. * means EGL w/o LU for LatentMAS-variant.

MAS WorkflowBaseline Acc. (%)LU Gain (pp)EGL Gain (pp)Combined Gain (pp)
VideoMind-style (Planner → Grounder → Verifier → Answerer)51.08+0.08+2.92+5.69
SVAgent-style(Storyline → Hypothesis → Meta-Decision → Refiner)51.76+0.22+1.87+6.58
LatentMAS-style(w. LU) (Planner → Critic → Refiner → Judger)55.65—+1.64+4.04

W1-3 Comparison with VideoMiner: ViLaMAS differs in two key aspects: (1) Evidence Localization: ViLaMAS targets visual evidence that distinguishes competing answer candidates after agent handoffs, rather than searching for question-relevant keyframes in a video hierarchy. (2) Evidence Integration: ViLaMAS incorporates newly retrieved frames into the inherited multimodal reasoning state through latent updates, enabling continuous reasoning and answer revision across agent handoffs.

We sincerely appreciate your constructive suggestion. As discussed above, our work offers practical engineering value while also making a fundamental methodological contribution to integrating visual evidence into latent reasoning.

AuthorsOpenReview ↗

Response to Reviewer h42S [2/4]

W #2 The proposed method can evidently only be applied to multi-agent scenarios with shared architecture and weights, imposing a significant limitation on generalization.

Response to W2.

Thank you for your suggestion. We clarify the framework's applicability from three perspectives:

W2-1 Backbone Flexibility: We acknowledge that Full-KV transfer requires compatible architectures and shared weights. However, this constraint applies to individual agent handoffs rather than the choice of backbone. ViLaMAS can be adapted to other compatible multimodal backbones. We conducted tests on different model backbones as shown in Table R4:

Table R4: Cross-Backbone Generalization of ViLaMAS on LongVideoBench.

BackboneSVI Acc. (%)Text Handoff Acc. (%)ViLaMAS Acc. (%)
Qwen3-VL-8B66.047.561.8
InternVL3-9B64.049.260.5
LLaVA-NeXT-Video-7B61.345.759.0

W2-2 Controlled Evaluation:

We deliberately adopt a shared backbone to isolate handoff-induced degradation from differences in model capability, enabling controlled evaluation of communication mechanisms.

W2-3 Practical Applicability:

Shared-backbone collaboration is itself a practical MAS setting, allowing specialized agents to operate through different role prompts while reusing a common model. Most mainstream MAS architectures adopt the same model architecture which simplifies deployment within compatible multi-agent systems.

W #3 The main experimental results (Table 1) show that the multi-agent variants (ViLaMAS) consistently underperform SVI. The paper does not clearly explain the rationale for adopting a multi-agent system at all, given that it fails to outperform the single-system SVI baseline. The motivation and added value of the multi-agent design remain unsubstantiated.

Response to W3:

We appreciate this concern and we will explain from motivation and value these two aspects:

W3-1 Motivation and Rationale: (1) Collaboration benefits in MAS and handoff losses are two distinct considerations in MAS and we focus on the latter: mitigate information loss when agent handoffs are part of an existing workflow. Such collaboration may be motivated by role specialization, independent verification, or tool integration, where preserving upstream visual evidence becomes essential.

W3-2 Design Value of MAS:

The value of MAS extends beyond standalone QA accuracy. MAS enables role specialization, modular extensibility, and collaborative reasoning, allowing complex tasks to be decomposed into specialized stages through flexible agent and tool integration. Recent surveys highlight the growing interest in multimodal agent systems [1-3]. However, effective collaboration relies on reliable information transfer across agents. ViLaMAS is specifically designed to address this critical bottleneck, enabling more reliable multi-agent collaboration.

We hope that it can promote the development of the multi-agent field. Thank you for your insights and suggestions sincerely.

[1] Xie et al. Large Multimodal Agents: A Survey. Visual Intelligence, 2025.

[2] Jiang et al. Towards Rationality in Language and Multimodal Agents: A Survey. NAACL, 2025.

[3] Chen et al. A Survey of Multi-Model Collaboration in Video Understanding. Data, 2026.

AuthorsOpenReview ↗

Response to Reviewer h42S [3/4]

Q #1 The paper needs a formal statement and proof of the equivalence between latent continuation after KV transfer and uninterrupted reasoning, and, where possible, an analysis of how error propagates (or is bounded) as handoff depth H grows.

Response to Q1:

Q1-1 Statement and Proof: Following your suggestion, we provide the formal statement and proof of the equivalence between latent continuation after KV transfer and uninterrupted reasoning as follows:

Proposition. Let Si=(Mi,πi)S _i=(M _i,\pi _i) denote the complete decoder-continuation state. Consider two executions of the same frozen decoder: uninterrupted continuation and continuation after KV transfer. Both append the same receiver prompt pjp _j and perform mjm _j latent updates using the fixed alignment map AθA _\theta (Eq. A.3).

Assume that (i) the transferred state is exactly restored, including all-layer KV tensors, cache positions, and multimodal positional metadata; (ii) model weights, prompts, attention rules, positional conventions, and latent-update schedules are identical; and (iii) computation is deterministic and performed in exact arithmetic.

Then, for every 0≤k≤mj0\leq k\leq m _j,

(h^{\mathrm{rest}} _{j,k},S^{\mathrm{rest}} _{j,k})=(h^{\mathrm{unint}} _{j,k},S^{\mathrm{unint}} _{j,k}). \tag{A.4}

Proof. Exact state restoration and identical deterministic prefill establish the base case:

(hj,0rest,Sj,0rest)=Prefill⁡θ(pj;Hi→j(Si))=Prefill⁡θ(pj;Si)=(hj,0unint,Sj,0unint).(h^{\mathrm{rest}} _{j,0},S^{\mathrm{rest}} _{j,0})=\operatorname{Prefill} _{\theta}(p _j;H _{i\to j}(S _i))=\operatorname{Prefill} _{\theta}(p _j;S _i)=(h^{\mathrm{unint}} _{j,0},S^{\mathrm{unint}} _{j,0}).

Suppose the states agree at step k<mjk<m _j. Since the alignment map AθA _\theta is fixed, both executions produce identical next latent embeddings:

Aθ(hj,krest)=Aθ(hj,kunint).A _\theta(h^{\mathrm{rest}} _{j,k})=A _\theta(h^{\mathrm{unint}} _{j,k}).

The deterministic decoder FθF _\theta therefore receives identical embeddings and continuation states, yielding

(hj,k+1rest,Sj,k+1rest)=Fθ(Aθ(hj,krest);Sj,krest)=Fθ(Aθ(hj,kunint);Sj,kunint)=(hj,k+1unint,Sj,k+1unint).(h^{\mathrm{rest}} _{j,k+1},S^{\mathrm{rest}} _{j,k+1})=F _\theta(A _\theta(h^{\mathrm{rest}} _{j,k});S^{\mathrm{rest}} _{j,k})=F _\theta(A _\theta(h^{\mathrm{unint}} _{j,k});S^{\mathrm{unint}} _{j,k})=(h^{\mathrm{unint}} _{j,k+1},S^{\mathrm{unint}} _{j,k+1}).

By induction, Eq. (A.4) holds for all latent updates. The same argument applies successively to HH exact handoffs along an identical sequentially inherited trajectory. This proposition makes explicit the assumptions and inductive argument underlying the continuation property in Appendix A.2.2 (Eq. A.4).

Q1-2 Analysis: Regarding error propagation with handoff depth HH, Figure 6 provides an empirical analysis. As HH increases from 0 to 6, textual handoff accuracy drops from 76% to 66%, whereas Full-KV + LU maintains approximately 75% accuracy. This demonstrates substantially reduced degradation across successive handoffs. To further validate state consistency across handoff depths, we extend our experiments by comparing KV-restored and uninterrupted execution (Table R5).

Table R5: Accuracy Retention and Numerical State Consistency Across Handoff Depths on a Video-MME subset.

Handoff Depth HHText Acc. (%)KV-only Acc. (%)KV+LU Acc. (%)Relative Hidden-State Error (10⁻⁶) ↓
07676760
17277771.8
27175762.5
46974752.3
66672752.1

Together, these analyses demonstrate that Full-KV + LU mitigates accuracy degradation as handoff depth increases (66% vs. 75% at H=6H=6), while providing a direct assessment of numerical state consistency across successive handoffs. We add this clarification in Appendix B (revised version).

Reviewer_hYszOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ViLaMAS, a training-free framework for mitigating performance degradation across agent handoffs in existing multi-agent system (MAS) workflows. Specifically, it first introduces Self-Verification Inference (SVI) as a baseline for long-video question answering. It then adds Reasoner and Aggregator roles after SVI and shows that continuing from the current computational state with appended role prompts is more effective than textual handoffs. Latent-state reasoning further recovers some of the performance lost during role transitions. Finally, EGL augments the final aggregator’s current state with additional evidence from the source video and produces a revised answer. Experiments on three video benchmarks show that ViLaMAS recovers part of the performance lost through agent handoffs.

Critical strengths

The degradation across agent handoffs addressed in this paper is an important issue for optimizing interactions within agent systems.

Critical weaknesses
  • The baseline is not well justified. Rather than evaluating the proposed approach within existing MAS workflows, the authors construct a workflow by adding Reasoner and Aggregator roles after SVI. However, this baseline workflow performs 18.5 percentage points worse than SVI, which substantially weakens the persuasiveness of the experimental setup.
  • The final method performs poorly and does not outperform SVI. What practical value does this workflow offer over a simpler alternative? Moreover, its agent modules and mechanisms lack novelty compared with existing agent methods.
Questions for authors
  • Regarding Weakness 1, can the proposed components be incorporated into existing agent methods and demonstrate performance improvements?
  • Regarding Weakness 2, can the final system outperform current state-of-the-art methods? If Full-KV amounts to continuing from the current state with a new role prompt, LU amounts to latent-state reasoning, and EGL appears to be a conventional revisiting module, what distinguishes this approach from existing agent methods?
5 comments
AuthorsOpenReview ↗

Response to Reviewer hYsz [1/3]

Thank you for your time and feedback, which helped us identify the points that require clarification.

W #1 The baseline is not well justified. Rather than evaluating the proposed approach within existing MAS workflows, the authors construct a workflow by adding Reasoner and Aggregator roles after SVI. However, this baseline workflow performs 18.5 percentage points worse than SVI, which substantially weakens the persuasiveness of the experimental setup.

Response to W1:

W1-1 Baseline Validity: We clarify that controlled role-configuration experiments (main paper) demonstrate comparable performance across different role-prompt configurations, not relying on a specific prompting setup. Matched-role and generic-role controls achieve comparable accuracy to ViLaMAS as shown in Table R1:

Table R1 : Role-configuration controls in Table 3 in main paper.

Configuration8B Acc. (%)32B Acc. (%)
ViLaMAS (Reasoner + Aggregator)61.864.1
Single instance (matched roles)61.463.6
Single instance (generic role)60.462.8

We would also like to clarify that agent handoff-induced degradation is a central focus of our work. The purpose of introducing Reasoner and Aggregator is to establish a controlled agent-handoff setting, allowing us to isolate the impact of communication mechanisms on final answer accuracy.

W1-2 Cross-Workflow Validity: To strengthen our experimental setup, we conduct additional controlled experiments using alternative role configurations adapted from existing MAS chain workflows to examine whether handoff-induced degradation persists across different roles (Table R2).

Table R2: Accuracy comparison across different post-SVI role configurations on LongVideoBench.

BackboneSVISVI → ReasonerSVI → RefinerSVI → VerifierSVI → AnswererSVI → Reasoner → AggregatorSVI → Refiner → Judger
8B Acc. (%)66.055.754.454.453.947.546.4
32B Acc. (%)70.260.459.959.558.751.450.1

The results show that handoff-induced degradation persists across different role configurations, further supporting the validity of our experimental setup.

Furthermore, beyond the chain architecture we mentioned, Figure 4 (main paper) also evaluates ViLaMAS across six representative MAS architectures (Table R3) demonstrating that its effectiveness is not confined to the constructed SVI-based workflow.

Table R3: Accuracy gap reduction across six MAS architectures on Perception Test from Figure 4 in main paper.

MAS ArchitectureText Gap (pp)Gap Reduction (%)
Star9.0367.1
Chain8.1873.5
Tree9.3164.9
Debate9.3155.2
Hypothesis–Verify7.6251.8
Storyline10.8649.6
AuthorsOpenReview ↗

Response to Reviewer hYsz [2/3]

W #2 The final method performs poorly and does not outperform SVI. What practical value does this workflow offer over a simpler alternative? Moreover, its agent modules and mechanisms lack novelty compared with existing agent methods.

Response to W2:

W2-1 Motivation: We would like to clarify that collaboration benefits and handoff losses are two distinct considerations in MAS, and we focus on mitigating the latter rather than designing a new collaborative workflow to outperform SVI. Reviewer xYmD and reviewer h42S's critical strengths reviews could clarify that through component-wise cumulative ablations starting from SVI, the paper reveals the marginal contribution of ViLaMAS.

W2-2 Practical Value: There are three aspects:

(1) ViLaMAS consistent accuracy recovery across six MAS architectures (Table R3) demonstrates its practical value when agent collaboration is required. Future multimodal MAS methods can benefit from integrating ViLaMAS to better preserve multimodal information across agent handoffs.

(2) Through controlled experiments, that text-based handoffs in multi-agent systems lead to progressive accuracy degradation, providing a valuable empirical observation for the field of multi-agent video understanding.

(3) ViLaMAS achieves effective recovery and improvement of accuracy while reducing output tokens, inference time, and FLOPs relative to textual handoffs. We additionally conduct a dedicated end-to-end inference cost comparison to demonstrate the computational efficiency (Table R4).

Table R4: End-to-end inference cost comparison on 100 Video-MME questions using Qwen3-VL-8B.

MethodAcc. (%) ↑FLOPs (TFLOPs) ↓E2E Latency (s) ↓
SVI7684.68526.880
ViLaMAS-Text68200.43635.753
ViLaMAS75160.58132.065

Experiments demonstrate that ViLaMAS improves accuracy while reducing FLOPs and end-to-end latency compared with textual handoffs (ViLaMAS-Text), highlighting its practical efficiency.

W2-3 Novelty:

From a methodological perspective, our methodological novelty lies in integrating visual evidence directly into inter-agent communication and reasoning through latent-state updates, without reducing visual information to textual representations. ViLaMAS localizes visual evidence based on competing answers in the inherited reasoning state, then appends the retrieved video observations to that state before performing further latent updates. This benefit offers a new perspective on integrating latent reasoning with visual evidence.

From a module perspective, (1) SVI integrates independent candidate generation, video-grounded self-verification, and evidence-aware answer fusion to improve the reliability of long-video understanding as a methodological innovation; (2) Latent reasoning preserves accumulated multimodal context across agent handoffs and enables continuous latent reasoning without reducing visual information to textual summaries; (3) EGL module is a state-preserving, closed-loop evidence updating mechanism designed for multi-agent video understanding.

To demonstrate EGL's distinctive advantages in multi-agent video understanding, we conduct additional controlled comparisons against alternative visual evidence localization strategies (Table R5).

Table R5: Comparison between different visual revisiting methods on Video-MME subset. Values are final accuracy (%).

Method300 questions (32B)
No revisiting69.33
Original-frame rereading70.00
Random eligible interval70.00
Original-question localization70.67
Evidence-guided localization (w/o LU)73.33
Evidence-guided localization (EGL)76.67

Experiments demonstrate that EGL offers a distinct approach while achieving substantial accuracy improvements over alternative revisiting strategies.

AuthorsOpenReview ↗

Response to Reviewer hYsz [3/3]

Q #1 Regarding Weakness 1, can the proposed components be incorporated into existing agent methods and demonstrate performance improvements?

Response to Q1:

Following your constructive suggestion, we conduct additional experiments by integrating our proposed modules into representative workflows adapted from existing agent methods to demonstrate their generalizability and effectiveness (Table R6).

Table R6: Transferability of ViLaMAS components to existing-method-inspired MAS workflows. LongVideoBench with Qwen3-VL-8B. Initial video inputs and role sequences are fixed within each workflow.

Base MethodConfigurationAccuracy (%)ΔAcc. (pp)
VideoMind-variantBaseline51.08—
( VideoMind-Style: Planner->Grounder->Verifier->Answerer)+ LU51.16+0.08
+ EGL54.00+2.92
+ EGL with LU56.77+5.69
SVAgent-variantBaseline51.76—
( SVAgent-Style: Storyline->Hypothesis->Decision(Meta)->Suggestion-guided refinement)+ LU51.98+0.22
+ EGL53.63+1.87
+ EGL with LU58.34+6.58
LatentMAS-variantBaseline55.65—
( LatentMAS-Style: Planner->Critic->Refiner->Judger)+EGL w/o LU57.29+1.64
+ EGL59.69+4.04

The results demonstrate that our proposed modules consistently improve accuracy across diverse existing-method-inspired MAS workflows, confirming their generalizability and practical applicability beyond the SVI-based setting. We have clarified this experiment in Appendix A.4 (revised version).

Q #2 Regarding Weakness 2, can the final system outperform current state-of-the-art methods? If Full-KV amounts to continuing from the current state with a new role prompt, LU amounts to latent-state reasoning, and EGL appears to be a conventional revisiting module, what distinguishes this approach from existing agent methods?

Response to Q2:

Q2-1 Performance: (1) We emphasize our work focuses on mitigating accuracy degradation caused by successive agent handoffs. Improving this communication bottleneck and mitigating accuracy degradation is an important contribution in its own right. (2) While surpassing all SOTA systems is not our primary objective, Table R6 demonstrates that ViLaMAS consistently improves accuracy across multiple MAS workflows adapted from existing methods. (3) Table R4 further shows that ViLaMAS achieves these accuracy gains while reducing computational costs, highlighting its "performance plus efficiency".

Q2-2 Distinguishing Mechanisms: Latent-space collaboration remains underexplored in multimodal MAS tasks. ViLaMAS distinguishes itself by introducing a visual evidence-guided latent update strategy that combines accuracy gains from visual evidence with computational efficiency from latent reasoning, offering complementary benefits across diverse MAS architectures (Table R3 and Table R5 and Table R6). Overall, its adaptability across MAS architectures, training-free nature, and novel perspective on integrating visual evidence into latent updates clearly distinguish it from existing agent methods. The controlled ablations and additional workflow adaptations (Table R6) demonstrate the benefits and transferability of this design.

We hope these clarifications and additional analyses address your concerns and encourage you to reconsider your overall assessment of our work.

Reviewer_xYmDOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces ViLaMAS, a training-free framework that reduces accuracy loss across agent handoffs by transferring complete multimodal KV states, continuing latent reasoning, and revisiting video evidence. It also introduces Self-Verification Inference (SVI) as a reference. With Qwen3-VL-8B on LongVideoBench, ViLaMAS improves accuracy from 47.5% with textual handoffs to 61.8% under the same workflow, compared with 66.0% before the added handoffs.

Critical strengths
  1. The controlled handoff experiments identify a meaningful failure in video-agent workflows and show that preserving context reduces the loss.
  2. Table 3 shows that latent updates help much more with KV transfer than with text alone, providing evidence that retained context and further reasoning work together.
Critical weaknesses
  1. LatentMAS already uses multi-layer KV transfer, latent updates, and the same ridge-alignment formula, while related visual-agent work also studies degradation across rounds. Table 1 includes prior methods, but their inputs and starting states are not clearly matched to the post-SVI controls in Table 3. A controlled comparison is needed to identify what the video-specific changes add beyond adapting these existing mechanisms.
  2. Sec. 3.3 generates a question to distinguish the top two answers and selects old frames by temporal proximity. If both answers share a wrong assumption, this search may miss evidence that challenges it; nearby frames may also miss a distant event needed for comparison. The reported gains support EGL, but do not establish how well it recovers in these cases.
  3. Secs. 3.2–3.3 retain prior reasoning and append new observations, but do not directly test whether clear new evidence can overturn a wrong intermediate answer. Appendix B.4 reports both corrections and harmful revisions, without isolating whether useful evidence was missing or was not used. Holding the new evidence fixed while varying the earlier hypothesis would clarify whether old reasoning limits correction.
Questions for authors
  1. Which changes beyond LatentMAS are needed for the proposed video setting? Can their effects be shown from the same SVI state, with matched inputs, roles, and latent-update budgets?
  2. How often does EGL recover the correct answer when it is outside the initial top two? Under the same budget, does a question aimed at challenging their shared assumption improve recovery?
  3. For questions linking distant events, would selecting old frames by their role in the question, rather than temporal proximity, improve evidence coverage?
  4. When the same clear evidence is supplied, how sensitive is the final answer to an earlier wrong hypothesis?
11 comments
AuthorsOpenReview ↗

Response to Reviewer xYmD [1/4]

We appreciate your comments on controlled comparisons and evidence utilization, which have helped us strengthen our experimental analysis.

W #1 LatentMAS already uses multi-layer KV transfer, latent updates, and the same ridge-alignment formula, while related visual-agent work also studies degradation across rounds. Table 1 includes prior methods, but their inputs and starting states are not clearly matched to the post-SVI controls in Table 3. A controlled comparison is needed to identify what the video-specific changes add beyond adapting these existing mechanisms.

Response to W1:

W1-1 Differences with LatentMAS: LatentMAS focuses on improving reasoning and communication efficiency among language agents, primarily evaluating mathematical and commonsense tasks, while ViLaMAS investigates how established multimodal understanding can be preserved, extended, and revised across successive agent handoffs in long-video reasoning.

W1-2 Differences with studies degradation across rounds: ViLaMAS integrates visual evidence directly into inter-agent communication and reasoning through latent-state updates, combining accuracy gains from visual evidence integration with the computational efficiency of latent reasoning.

W1-3 Controlled comparison: Following your constructive suggestions, we conducted three additional experiments:

Experiment 1: A comparison between LatentMAS and ViLaMAS further validates the method comparison under same inputs and starting states (Table R1). This experiment evaluates the additional benefits of our video-specific extensions.

Table R1: Comparison on LongVideoBench with Qwen3-VL-8B/32B. LatentMAS (SVI-based) denotes our controlled best-effort multimodal adaptation initialized from the SVI fusion state. Matched inputs, roles, and latent-update budgets.

Configuration8B Acc. (%)32B Acc. (%)
SVI (Reference)66.070.2
SVI → A1 (Text)55.760.4
SVI → A1 → A2 (Text)47.551.4
LatentMAS (post-SVI-based adaptation)*47.652.2
Full-KV + LU (w/o EGL)58.961.0
ViLaMAS (Full-KV + LU + EGL)61.864.1

*: The original LatentMAS framework was evaluated only on language-based tasks and the value in Table 1 in manuscript is not SVI-based.

Experiment 2: We conduct controlled comparisons to further evaluate the generalizability of LU and EGL across existing MAS workflows. LU alone yields only marginal improvements across different MAS architectures. In contrast, selectively incorporating relevant visual evidence into latent updates not only enhances multimodal information preservation but also unlocks the potential of LU for mitigating handoff-induced information loss.

Table R2: Generalizability of modules across existing methods' MAS workflow.

MAS WorkflowBaseline Acc. (%)LU Gain (pp)EGL Gain (pp)Combined Gain (pp)
VideoMind-style51.08+0.08+2.92+5.69
SVAgent-style51.76+0.22+1.87+6.58
LatentMAS-style(w. LU)55.65—+1.64*+4.04

*: +1.64 pp denotes EGL without additional post-revisit latent updates.

Experiment 3: Fix the existing communication and inference mechanisms of ViLaMAS, only changing the way visual evidence is obtained and integrated. Same SVI-based, model, roles and prompts. For revising configurations, use the same number of newly added frames, maximum revisit times, and post revisit LU budget.

Table R3: Controlled Ablation of Video-Specific Evidence Acquisition on LongVideoBench.

ConfigurationAdditional Visual Evidence8B Acc. (%)32B Acc. (%)
Full-KVNone52.255.9
Random Visual Revisiting with LURandom eligible intervals59.861.5
State-Conditioned EGL with LU (Ours)Candidate-discriminating evidence61.864.1

These findings support the distinct contribution and generalizability of our video-specific extensions. We sincerely appreciate your constructive suggestions, which we believe have helped strengthen the completeness and rigor of our work. We revised these experiments in Section 4.2.3 and Appendix B.4 (revised version).

AuthorsOpenReview ↗

Response to Reviewer xYmD [2/4]

W #2 Sec. 3.3 generates a question to distinguish the top two answers and selects old frames by temporal proximity. If both answers share a wrong assumption, this search may miss evidence that challenges it; nearby frames may also miss a distant event needed for comparison. The reported gains support EGL, but do not establish how well it recovers in these cases.

Response to W2 :

W2-1 Clarification: We clarify that Top-2 candidates guide evidence localization rather than restrict the final answer space. After incorporating new visual evidence, EGL re-evaluates all answer candidates, allowing the model to revise its initial hypotheses and consider alternatives beyond the Top-2.

W2-2 Beyond Top-2 Experiment: Following your constructive suggestion, we conduct a controlled experiment to evaluate EGL's ability to recover correct answers outside the initial Top-2 and whether challenging shared assumptions improves recovery (Table R4). We compare standard EGL with a shared-assumption challenging strategy under identical visual revisiting and latent-update budgets.

Table R4: Answer recovery beyond the initial Top-2 on 40 qualified LongVideoBench questions under matched inference budgets. Not all questions shared incorrect assumption.

MethodSubsetRecovery outside Top-2 (%) ↑Evidence-Supported Recovery (%) ↑
Standard EGL (Ours)All edge cases22.5 (9/40)66.7 (6/9)
Standard EGL (Ours)Shared-assumption29.4 (5/17)40.0 (2/5)
Shared-assumption ChallengingShared-assumption64.7 (11/17)27.3 (3/11)

The results indicate that EGL has limited recovery capability in these challenging cases. In contrast, shared-assumption challenging improves nominal recovery but yields a lower evidence-supported recovery rate.

W2-3 Distant-Event Experiment: We compare our current proximity-based selection with a question-role-guided strategy that selects original frames according to their relevance to different events (e.g., before/after or cause/effect; the event interval exceeds 60 seconds), under identical frame and LU budgets (Table R5).

Table R5: Comparison of original-frame selection strategies on temporally distant-event questions. Revisited Evidence Coverage measures the proportion of question-relevant events covered by the revisited frames.

MethodRevisited Evidence Coverage (%) ↑Accuracy (%) ↑
Temporal Proximity (Ours)27.344.4
Question-Role-Guided35.047.2

These results indicate that question-role-guided selection can improve coverage of temporally distant evidence. We further investigate whether we could alleviate the loss of such long-distance events by changing the number of original frames per revisit in EGL. As shown in Table R6, the improvements are modest, but long distant events remain an important exploration perspective.

Table R6: Sensitivity to the Number of Original Frames for long distant event.

Original FramesNew FramesEvidence Coverage (%)Accuracy (%)
4427.344.4
8427.744.9
12428.246.1

We revised the manuscript to include these experiments in Section 4.2.3 (revised version).

AuthorsOpenReview ↗

Response to Reviewer xYmD [3/4]

W #3 Secs. 3.2–3.3 retain prior reasoning and append new observations, but do not directly test whether clear new evidence can overturn a wrong intermediate answer. Appendix B.4 reports both corrections and harmful revisions, without isolating whether useful evidence was missing or was not used. Holding the new evidence fixed while varying the earlier hypothesis would clarify whether old reasoning limits correction.

Response to W3:

W3-1 Prior-Hypothesis Sensitivity: Thank you for your valuable experimental suggestion, which helps strengthen the rigor and completeness of our evaluation of evidence utilization. To investigate this, we conduct a Fixed-Evidence Hypothesis Intervention experiment (Table R7), providing identical, manually verified visual evidence (200 LongVideoBench questions) while appending incorrect or neutral hypothesis prompts to the same inherited KV state and independently continuing inference under each condition.

Table R7: Controlled Prior-Hypothesis Sensitivity Analysis under identical corrective visual evidence on a manually verified LongVideoBench subset. We inject hypotheses from the same KV checkpoint and provide identical visual evidence under matched inference budgets to isolate the effect of prior hypotheses.

Prior HypothesisWithout Evidence Acc.(%) (8B)↑After-Evidence Acc. (%) (8B)↑Acc. Gap vs. Neutral (pp)Without Evidence Acc.(%) (32B)↑After-Evidence Acc. (%) (32B)↑Acc. Gap vs. Neutral (pp)
Neutral Prior55.563.5058.067.00
Incorrect Prior34.057.5-6.037.562.5-4.5

With identical corrective visual evidence, accuracy under incorrect priors improves by 23.5 pp (8B) and 25.0 pp (32B), narrowing the gap to neutral priors to 6.0 and 4.5 pp, respectively. These results demonstrate that visual-to-latent continuation can substantially mitigate the influence of incorrect prior hypotheses when sufficient corrective evidence is available.

W3-2 Evidence Acquisition and Utilization Analysis:

To investigate the method's ability to utilize corrective visual evidence, we compare standard EGL with oracle-evidence revisiting using manually verified corrective frames, starting from the same naturally incorrect intermediate states (Table R8).

Table R8: Controlled comparison of answer correction using automatically retrieved versus oracle visual evidence on LongVideoBench. All samples have sufficient error correction evidence, and the error correction rate may be higher than full set.

ConfigurationVisual EvidenceWrong→Correct Rate
Random revisitingrandom3.5%
Standard EGLAutomatically localized evidence31.0%
Oracle-Evidence RevisitingManually verified corrective frames66.5%

These results suggest that automatic evidence acquisition remains an important bottleneck, while evidence utilization also leaves room for improvement. Overall, these results demonstrate that visual-to-latent continuation can revise incorrect prior hypotheses when sufficient corrective evidence is available.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.