acceptodds
Under review as a conference paper at ICLR 2027

Beyond Reasoning: Consensus-Guided Visual Collaboration in Latent Multi-Agent Systems

Abstract

Visual multi-agent systems (VMAS) leverage complementary reasoning roles for complex vision-centric tasks, but their prevalent reliance on natural-language communication creates an information bottleneck and substantial inference overhead. To address these limitations, we instantiate latent collaboration for VMAS, where agents reuse a shared visual KV cache and propagate intermediate reasoning KV states without generating textual messages. However, reasoning-state collaboration alone does not explicitly coordinate the role-conditioned visual attention patterns over the shared visual input. Our analysis shows that visual regions jointly prioritized across agents exhibit higher ground-truth region hit rates, suggesting cross-agent visual attention as a complementary signal for collaboration. Motivated by this observation, we propose V-Guide, a training-free framework that augments latent reasoning-state collaboration with explicit cross-agent visual coordination. Specifically, V-Guide first spatially marginalizes role-conditioned visual attention distributions to accommodate local token-level discrepancies, and then aggregates them into a geometric consensus distribution that captures cross-agent support. Finally, the resulting consensus distribution modulates the shared visual Value states, increasing the contribution of consensus-supported visual content for final reasoning. Extensive experiments across five visual reasoning benchmarks, four MLLM backbones, and different multi-agent architectures demonstrate that V-Guide outperforms text-mediated and pure-latent multi-agent baselines while retaining the efficiency advantages of latent collaboration.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Visual Multi Agent System, Multimodal Large Language Model, Visual Collaboration

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_XoNxOpenReview ↗
Rating
3
Confidence
2(−1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies latent collaboration in visual multi-agent systems and proposes V-Guide, a training-free framework. Agents can use a shared visual KV cache and communicate through latent reasoning states. V-Guide additionally extracts visual attention from each agent’s textual-prefix queries, applies Gaussian spatial smoothing, and constructs a geometric consensus distribution. This distribution modulates visual Value states during final answer generation. Experiments cover four multimodal backbones, five benchmarks, and sequential and hierarchical collaboration architectures. The results show consistent accuracy improvements over pure latent collaboration while retaining efficiency advantages over the evaluated text-mediated systems.

Critical strengths
  1. The improvements are consistent across backbones and collaboration architectures. As shown in table 1 and 5 covering three Qwen3-VL scales and InternVL3-8B, V-Guide improves all five benchmark scores over the corresponding Latent-VMAS baseline in every configuration. This consistency strengthens confidence in the method’s effectiveness and transferability.
  2. The training-free design has practical value. V-Guide requires neither an additional trained module nor extra visual tokens. It operates through visual-cache reuse, existing query states, and Value modulation. Figure 3 reports 1.7×–4.4× end-to-end speedups and 32%–66% fewer generated tokens relative to the implemented Text-VMAS, while maintaining efficiency comparable to Latent-VMAS.
Critical weaknesses
  1. The contributions of role specialization and latent reasoning are insufficiently isolated. Section 3.3 constructs agent attention distributions from textual-prefix queries rather than directly aggregating queries from latent reasoning steps. Although later prefixes can incorporate preceding agents’ reasoning, all agents share the same backbone and question. The ablations lack identical-role, redundant-role, or disabled-latent-reasoning controls. Consequently, the results do not clearly distinguish complementary-agent collaboration from aggregating correlated prompt-conditioned attention maps.
  2. Geometric consensus favors regions jointly attended to by multiple agents, while a low probability assigned by one agent reduces the corresponding consensus weight. The paper does not analyze cases where relevant evidence receives support from only a minority of agents, leaving unclear whether the mechanism adequately exploits specialist-specific information.
Questions for authors
  1. With comparable agent counts and computational settings, how does performance change when agents use identical role prompts, or when latent reasoning is disabled while prefix-attention aggregation is retained? Can these controls separate the contributions of role complementarity and latent reasoning?
  2. How does V-Guide perform when the decisive visual region is primarily attended to by only one agent?
1 comment
AuthorsOpenReview ↗

Point-to-Point Response to Reviewer XoNx

We’re delighted that you appreciated V-Guide’s consistent improvements, training-free design, and efficiency advantages! Thank you for your thoughtful and encouraging review. We provide point-by-point responses below.

W1&Q1: Isolating the Contributions of Role Specialization and Latent Reasoning.

A1: Insightful comment!

i. Contribution of Role Specialization.

To isolate the contribution of role specialization, we compare three configurations: Distinct Roles (Ours), where agents perform complementary reasoning roles; Identical Roles, where intermediate agents use identical role prompts; and Paraphrased Same Role, where they share the same semantic role but use differently worded prompts. Only the intermediate agents' role instructions are varied, with all other settings held constant. We further evaluate each configuration with and without consensus-guided visual Value modulation.

Role ConfigurationVStar w/oVStar w/GainRealWorldQA w/oRealWorldQA w/Gain
Distinct Roles (Ours)79.5885.34+5.7670.5973.86+3.27
Identical Roles78.0179.58+1.5769.2870.98+1.70
Paraphrased Same Role78.0180.10+2.0969.9371.63+1.70

As shown above, (1) Role specialization improves overall performance. Distinct Roles consistently outperform both same-role configurations, while identical and paraphrased same-role prompts achieve comparable accuracy, suggesting that the improvements cannot be explained by surface-level prompt variations alone. (2) Visual consensus is more effective with distinct roles. Incorporating visual consensus yields larger improvements with Distinct Roles (+5.76 on VStar and +3.27 on RealWorldQA), compared with +1.57–2.09 on VStar and +1.70 on RealWorldQA under the same-role configurations.

ii. Contribution of Latent Reasoning.

To isolate the contribution of latent reasoning, we introduce a No Latent Reasoning baseline that disables continuous latent reasoning (T=0T=0) while retaining textual-prefix KV propagation, prefix-attention aggregation, and consensus-guided visual Value modulation. V-Guide outperforms this baseline on VStar and RealWorldQA, indicating that continuous latent reasoning provides additional benefits.

MethodVStarRealWorldQA
No Latent Reasoning (T=0)84.8271.63
V-Guide (T=20)85.3473.86

W2&Q2: Performance under Single-Agent Visual Support.

A2: Thoughtful question! To examine whether geometric consensus harms cases where the target visual evidence is supported by only one agent, we conduct an additional analysis on VStar using Qwen3-VL-8B-Instruct. We select VStar because it provides ground-truth (GT) bounding boxes for evaluating visual localization. For each agent, we identify its top-3 attended regions from the visual attention distribution in Eq. (1), following the localization criterion in Appendix E.1. We focus on samples where exactly one of the four agents successfully localizes the GT region.

Among the 191 VStar samples, 12 (6.28%) satisfy this criterion. We compare Latent-VMAS and V-Guide on these same samples:

GT SupportSamplesWrong → CorrectCorrect → WrongNet Gain
Single-Agent Support12 (6.28%)5 (41.67%)1 (8.33%)+33.33

V-Guide corrects 5 predictions that are incorrect under Latent-VMAS, while only 1 correct prediction becomes incorrect, yielding a net improvement of +33.33 percentage points on these 12 samples. Although this subset is small, the results show that V-Guide can improve prediction accuracy even when only one agent successfully localizes the GT region.

This observation is consistent with our formulation. While geometric consensus assigns lower relative weights to regions with limited cross-agent support, it does not remove their visual information. Specifically, Eq. (4) applies bounded Value modulation with 1≤1+λwj≤1+λ1\leq1+\lambda w_j\leq1+\lambda, while leaving the visual Keys unchanged. Thus, minority-supported visual tokens may receive less enhancement but remain accessible to the final answering agent.

Together, these results suggest that consensus-guided modulation can remain beneficial even when the GT region is localized by only one agent, rather than necessarily suppressing such cases. We have incorporated this additional analysis and discussion into Appendix C.5 of the revised manuscript.


Thank you again for your thoughtful review! We hope our responses have addressed your concerns and would be happy to provide further clarification if needed.

Reviewer_pCUROpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper identifies a meaningful weakness in latent visual multi-agent collaboration: Latent VMAS coordinates agents' reasoning states, but it does not coordinate what different agents visually attend to. The paper proposes a coherent solution called V-Guide, and validates the proposed mechanism reasonably well.

Critical strengths
  1. The proposed method is training-free and effective. V-Guide improves over the latent-collaboration baseline while largely preserving its efficiency advantage, and is substantially cheaper than text-mediated VMAS.

  2. The empirical analysis about "why it works" is good and elegant.

Critical weaknesses

My primary concern is whether the current experiments sufficiently isolate the source of the gains attributed to role-conditioned cross-agent visual consensus. In the introduction, the authors state that "This produces distinct role-conditioned visual attention distributions over the shared visual tokens across agents" and "we hypothesize that cross-agent visual attention may provide complementary information or visual collaboration in VMAS". The paper proposes V-Guide based on these insights and attempts to validate it through empirical evidence. However, there are two gaps between the conceptual goal and the implemented algorithms:

  1. The paper attributes the attention diversity to the complementary roles of different agents, but it is unclear whether this diversity is truly induced by semantic role specialization or simply by differences in their prompts and contexts. How essential are the distinct agent roles to the effectiveness of the proposed consensus?

  2. The paper further attributes the gains to multi-agent visual collaboration, but it is unclear whether the benefit really comes from inter-agent collaboration or mainly from aggregating multiple role-conditioned attention patterns. How essential is the multi-agent interaction itself to the observed improvement?

Questions for authors

See the weaknesses above.

One more question: Specifically, the Gaussian kernel in Eq. 2 only guarantees: $ \sum_j G_\sigma(j \mid k) = 1, $ without ensuring: $ \sum_k G_\sigma(j \mid k) = 1. $ Consequently, the transition operator is column-stochastic, but not necessarily doubly stochastic. On a bounded, finite discrete grid, normalization constants differ across source positions due to boundary truncation effects. As a result, an uninformative, strictly uniform input distribution is not guaranteed to remain uniform after transformation, potentially introducing an artificial spatial bias even when the model exhibits zero visual attention preference. How do the authors address this boundary artifact, and to what extent does this artificial spatial prior contribute to the empirical consensus gains reported in the benchmarks?

2 comments
AuthorsOpenReview ↗

Point-to-Point Response to Reviewer pCUR [1/2]

We're truly grateful that you recognized the effectiveness and efficiency of our training-free V-Guide framework, as well as the clarity and elegance of our empirical analysis! Thank you for your thoughtful and constructive review. We provide point-by-point responses below.

W1: How Essential Are Distinct Agent Roles to Visual Consensus?

A1: Insightful comment! In V-Guide, semantic roles are specified through role-specific prompts, while accumulated latent contexts further influence the resulting visual attention. To investigate the contribution of distinct agent roles to visual consensus, we evaluate three role configurations, each with and without Consensus-Guided Visual Value Modulation:

  1. Distinct Roles: Intermediate agents perform distinct reasoning roles.
  2. Identical Roles: Intermediate agents share the same semantic role and identical prompt.
  3. Paraphrased Same Role: Intermediate agents share the same semantic role but use differently worded prompts.

All configurations use the same Qwen3-VL-8B-Instruct backbone, agent count, latent reasoning steps, inter-agent communication mechanism, and final answering agent.

Role ConfigurationVStar w/oVStar w/GainRealWorldQA w/oRealWorldQA w/Gain
Distinct Roles79.5885.34+5.7670.5973.86+3.27
Identical Roles78.0179.58+1.5769.2870.98+1.70
Paraphrased Same Role78.0180.10+2.0969.9371.63+1.70

(i) Surface-level prompt variations cannot explain the benefits of distinct roles. With visual consensus enabled, paraphrasing the same role yields only small accuracy differences compared with identical prompts (+0.52 on VStar). In contrast, distinct roles achieve substantially higher accuracy.

(ii) Visual consensus is more effective with distinct semantic roles. Incorporating consensus-guided modulation yields improvements of +5.76 and +3.27 points on VStar and RealWorldQA with distinct roles, respectively, compared with +1.57/+2.09 and +1.70/+1.70 points under the two same-role configurations.

We have incorporated these additional results and discussions into Appendix B.

W2: Contribution of Inter-Agent Interaction.

A2: Thoughtful question! To isolate the contribution of inter-agent latent interaction, we introduce an Independent Attention Consensus baseline, where agents independently compute role-conditioned visual attention without inter-agent KV propagation, while retaining the same visual consensus and Value modulation mechanisms as V-Guide. All other settings remain consistent.

MethodInter-Agent Latent CollaborationConsensus-Guided Visual Value ModulationVStarRealWorldQA
Single✗✗83.7771.11
Latent-VMAS✓✗79.5870.59
Independent Attention Consensus✗✓84.2971.24
V-Guide✓✓85.3473.86

(i) Independent attention aggregation alone cannot fully explain V-Guide's improvements. With the same consensus-guided visual Value modulation, V-Guide outperforms Independent Attention Consensus by +2.62 on RealWorldQA, demonstrating the additional contribution of inter-agent latent interaction.

(ii) Consensus-guided visual Value modulation substantially improves latent collaboration. With inter-agent latent interaction unchanged, adding our proposed modulation improves Latent-VMAS by +5.76 on VStar and +3.27 on RealWorldQA, demonstrating the benefit of consensus-guided visual Value modulation for latent collaboration.

Overall, these results show that independent attention aggregation alone cannot match V-Guide, while consensus-guided modulation substantially improves latent collaboration. The two mechanisms therefore provide complementary contributions to V-Guide's performance. We have incorporated these additional results and discussions into Appendix B.

AuthorsOpenReview ↗

Point-to-Point Response to Reviewer pCUR [2/2]

Q1: Boundary Artifacts of Gaussian Spatial Marginalization.

A3: Thanks for your careful review! We address the potential boundary-induced spatial bias in Eq. (2) through a doubly-stochastic Gaussian kernel and evaluate its impact on consensus performance.

Addressing the Boundary Artifact. We apply symmetric Sinkhorn normalization to the Gaussian affinity matrix AA, obtaining G^σ=DAD\widehat{G}_\sigma=DAD, where DD is a positive diagonal scaling matrix. The resulting kernel is doubly stochastic, preserving uniform attention distributions while retaining Gaussian spatial smoothing and the probability-preservation and contraction properties established in Appendix F.2.

Impact on Empirical Consensus Gains. We evaluate V-Guide with the boundary-corrected kernel, keeping all other settings unchanged.

MethodRealWorldQAVStar
Latent-VMAS70.5979.58
V-Guide (Original Gaussian)73.8685.34
V-Guide (Doubly-Stochastic Gaussian)73.7385.86

The corrected kernel achieves comparable performance to the original. It also retains substantial improvements of +3.14/+6.28 points over Latent-VMAS. These results demonstrate that V-Guide's consensus gains persist after eliminating the boundary-induced bias for uniform attention, indicating that the gains do not rely on this artifact. We have incorporated the boundary-corrected formulation and experimental results into Appendix F.2 of the revised manuscript.


Thank you again for your thoughtful review! We hope our responses have addressed your concerns and would be happy to provide further clarification if needed.

Reviewer_6aVXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes V-Guide, a training-free latent collaboration framework for Visual Multi-Agent Systems (VMAS). To overcome the information bottleneck and decoding latency of text-mediated multi-agent communication (Text-VMAS), the authors first instantiate Latent-VMAS, where agents share a single-pass visual KV cache and pass intermediate latent reasoning KV states. Observing that Latent-VMAS does not explicitly coordinate where different agents look in the image—and that visual tokens attended to by multiple agents exhibit higher ground-truth region hit rates—the authors introduce three training-free operations for final answer generation. Experiments across 5 benchmarks and 4 MLLM backbones show average-score gains over previous approaches in the reported configurations.

Critical strengths
  1. This paper has a clean training-free formulation for visual consensus in latent VMAS to adapt continuous latent KV-cache communication to multimodal vision-language reasoning.
  2. V-Guide consistently repairs the performance degradation of Latent-VMAS on VStar and RealWorldQA and achieves higher reported average scores than the compared baselines.
Critical weaknesses
  1. The proposed VMAS is not significantly different from text-based multi-agent systems. Why using visual KV-cache is important and necessary compared with using text description? The proposed approach is tricky and lacks theoretical motivation.
  2. The visual attention extraction and latent reasoning trajectories may be disconnected. Each agent AiA_i's visual attention distribution is computed by mean-pooling the Query states of its textual prefix (the question and role instruction), not from the Query states generated during its T=20 continuous latent reasoning steps.
  3. Most gains come from single-agent value scaling rather than cross-agent consensus in the ablation table. Worse still, the paper omits some critical baselines such as Single + Self-Attention Value Modulation.
Questions for authors

See the weaknesses above.

2 comments
AuthorsOpenReview ↗

Point-to-Point Response to Reviewer 6aVX [1/2]

Thanks for your encouraging feedback on our efficient, training-free visual collaboration and strong performance gains! We address your concerns below.

W1: Distinction and Motivation.

A1: Sorry for the confusion! We clarify the fundamental differences in collaboration paradigms, the rationale for shared visual KV caching, and the motivation underlying our technical designs.

(1) Different Collaboration Paradigms. Conventional text-mediated collaboration relies on explicit natural-language reasoning and communication. In contrast, V-Guide explores pure-latent collaboration for visual reasoning, where agents perform continuous latent reasoning and communicate directly through KV states without intermediate text generation. This represents a fundamental difference in both reasoning and communication. Although latent collaboration has shown promise in language-based MAS [18–21], its extension to vision-centric reasoning remains underexplored.

(2) Why Shared Visual KV Cache Rather Than Text Descriptions?

  • Avoiding Information Loss from Verbalization: Text-mediated collaboration requires visual observations and intermediate reasoning to be verbalized, potentially discarding fine-grained information or prematurely committing to uncertain interpretations [12,15,18]. In contrast, V-Guide propagates intermediate reasoning through latent KV states while retaining direct access to the shared visual representation.

  • Mitigating Text-Mediated Error Propagation: Prior work [13] has identified hallucination snowballing caused by the propagation of erroneous textual information across visual agents. By communicating through latent KV states rather than intermediate text descriptions, V-Guide reduces reliance on potentially inaccurate verbalized reasoning.

  • Direct Visual Access and Coordination: Shared visual Keys establish a common visual-token representation across agents, allowing their role-conditioned attention distributions to be directly compared and aggregated without explicit textual grounding.

  • Efficiency: Reusing visual KV caches avoids repeated image encoding, while latent communication eliminates intermediate text generation and processing. As shown in Fig. 3, V-Guide achieves 1.7×–4.4× inference speedup and 32%–66% lower generated-token cost compared with Text-VMAS.

(3) Clear Motivation and Principled Technical Design.

V-Guide addresses a fundamental limitation of existing latent multi-agent collaboration, motivated by a key empirical insight and developed through a coherent sequence of mathematically grounded designs.

i. Motivation and Key Insight.

Existing latent multi-agent frameworks enable efficient reasoning-state communication but lack explicit coordination of visual evidence across agents. In Table 1, Latent-VMAS can even underperform single-agent baselines on vision-centric tasks, suggesting that reasoning-state exchange alone does not guarantee effective visual collaboration.

To address this limitation, we investigate whether cross-agent visual attention can provide a useful signal for visual coordination. Our analysis reveals that visual regions jointly prioritized by multiple agents exhibit substantially higher ground-truth region hit rates (Fig. 1(a)). This insight motivates V-Guide to leverage cross-agent visual attention consensus to guide final reasoning, complementing reasoning-state exchange with explicit visual coordination.

ii. Challenges.

Realizing this insight presents two challenges:

  • C1: Agents may attend to slightly different locations within the same relevant region, requiring spatially tolerant yet agreement-sensitive consensus estimation.
  • C2: Cross-agent consensus must effectively guide final reasoning while preserving training-free and efficient latent collaboration.

iii. Principled Technical Designs.

V-Guide introduces two complementary, mathematically grounded designs:

  • To tackle C1, we introduce Spatial Uncertainty Marginalization to accommodate local attention discrepancies, with provable L1 and KL non-expansiveness (App.F.2). We further derive Geometric Consensus as the closed-form solution to a reverse-KL minimization objective, emphasizing consistent cross-agent support rather than simply averaging attention (Eq.3; App.F.3).

  • To tackle C2, we propose Consensus-Guided Visual Value Modulation to reinforce consensus-supported visual content without additional training. Its effect on attention outputs is mathematically bounded, while preserving attention logits for a fixed incoming query (Eqs.4–5).

Overall, V-Guide is not a collection of heuristic tricks, but a principled solution to an important yet underexplored problem: coordinating visual evidence across agents in latent multi-agent reasoning. Its designs follow directly from the identified challenges, supported by mathematical analysis and empirical validation.

AuthorsOpenReview ↗

Point-to-Point Response to Reviewer 6aVX [2/2]

W2: Visual Attention Extraction and Its Connection to Latent Reasoning.

A2: Thoughtful question! We clarify that the prefix-based visual attention readout is not entirely disconnected from latent reasoning. For agents receiving collaborative latent memory from preceding agents, their textual-prefix Query states are computed in the context of these propagated KV states and can therefore reflect prior latent reasoning. Importantly, V-Guide aims to extract role-conditioned, question-relevant visual attention patterns for cross-agent consensus, rather than explicitly tracking visual attention at every latent reasoning step.

To directly investigate this concern, we conduct an additional Query-source ablation, replacing the textual-prefix Query states of the intermediate agents with their latent reasoning Query states, while retaining the answering agent's prefix-based attention. All other components, including the four-agent consensus, latent-state collaboration, and visual Value modulation, remain unchanged.

Query SourceVStarRealWorldQA
Latent-VMAS (No Consensus)79.5870.59
Prefix Query (Original)85.3473.86
Latent Query85.8673.46

These results show that both prefix-derived and latent reasoning-derived attention can effectively support cross-agent visual consensus, indicating that the benefits of visual coordination do not depend on restricting attention extraction to textual-prefix states. We have incorporated this additional ablation into Section 4.4.

W3: Source of Performance Gains and Missing Single-Agent Baseline.

A3: To directly address this concern, we have conducted additional experiments with Single + Self-Attention Value Modulation baseline using Qwen3-VL-8B-Instruct. This baseline applies the same visual Value modulation formulation as Eq. (4), but derives its modulation weights solely from a single agent's visual attention, without cross-agent consensus.

We report the new results alongside the controlled visual modulation comparisons from Table 4.

MethodMMStarRealWorldQAVStar
Single67.0771.1183.77
Single + Self-Attention Value Modulation67.0070.0783.25
Latent-VMAS68.0070.5979.58
Latent-VMAS + Uniform Weighting70.1370.3379.58
Latent-VMAS + Final-Agent Attention70.3370.3383.77
V-Guide (Cross-Agent Consensus)72.0773.8685.34

The results provide two important observations:

  • Self-Attention Value Modulation Does Not Improve Single-Agent Performance. Compared with the Single baseline, Single + Self-Attention Value Modulation yields changes of −0.07/−1.04/−0.52 points on MMStar/RealWorldQA/VStar, respectively. These results indicate that applying attention-guided visual Value modulation alone does not guarantee performance improvements.

  • Cross-Agent Consensus Further Improves upon Final-Agent Attention Guidance. While final-agent attention-guided modulation already provides gains on MMStar and VStar, cross-agent consensus consistently improves upon it under the same latent collaboration framework and modulation formulation. Notably, on RealWorldQA, final-agent guidance slightly decreases accuracy, whereas cross-agent consensus improves it by 3.27 points over Latent-VMAS. These results demonstrate that cross-agent consensus provides additional benefits that cannot be explained by final-agent attention guidance alone.

Furthermore, the Uniform Weighting baseline in Table 4 matches the average modulation strength of V-Guide but consistently underperforms our method, further indicating that the improvements cannot be explained solely by increasing visual Value magnitudes.


We sincerely appreciate your constructive feedback and hope we addressed all your concerns. Please let us know if you'd like any further information.

Reviewer_G1RzOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents V-Guide, a training-free framework for coordinating visual attention in visual multi-agent systems (VMAS). It builds on a latent communication setup in which role-specialized agents share a visual key–value (KV) cache computed in a single pass and exchange continuous reasoning states without decoding intermediate text. The authors find that differences in where agents attend can impair fine-grained visual grounding. V-Guide addresses this by extracting each agent’s role-conditioned visual attention distribution from intermediate layers and applying a two-dimensional Gaussian transition kernel to account for small spatial shifts between visual tokens. It then uses reverse KL divergence to compute a geometric consensus distribution that reweights the shared visual value states during final answer generation. Across five multimodal reasoning benchmarks, four multimodal large language model backbones, and two agent topologies (sequential and hierarchical), V-Guide consistently improves accuracy over single-agent, text-mediated, and pure-latent baselines. Compared with text-mediated collaboration, it achieves a 1.7×–4.4× inference speedup and uses 32%–66% fewer tokens.

Critical strengths

1- Training-free coordination of visual attention across agents. V-Guide pools role-conditioned visual attention from intermediate-layer queries, uses a 2D Gaussian kernel to account for small shifts on the visual token grid, and derives a shared attention distribution through reverse-KL geometric consensus. This gives agents with different roles a principled way to agree on task-relevant regions without additional training or explicit bounding boxes.

2- Accuracy gains while preserving inference efficiency. The paper identifies a limitation of Latent-VMAS: it can underperform single-agent inference on fine-grained spatial reasoning and visual grounding benchmarks such as VStar and RealWorldQA. V-Guide improves average accuracy over Latent-VMAS by up to 8.36 percentage points while retaining the latency and token savings of continuous latent communication.

Critical weaknesses
  1. KV-cache aggregation in the hierarchical architecture. The paper provides limited detail on how it merges the KV caches produced by parallel agents. Please clarify how positional indices are assigned to intermediate latent states when the branches are combined. If cached keys from different branches reuse the same RoPE position IDs, explain how this affects positional information in the downstream attention computation.

  2. Shared backbone and correlated perceptual biases. Direct KV-cache sharing and visual token alignment require agents to use the same frozen MLLM backbone, which limits the use of heterogeneous model ensembles. A shared backbone may also induce correlated perceptual errors. For example, agents with different roles may still focus on the same misleading region, and the consensus mechanism could reinforce that error.

  3. Limited comparisons with vision-specific multi-agent methods. Although V-Guide builds on LatentMAS for latent KV-cache propagation, the evaluation mainly compares it with general-purpose multi-agent protocols. Including vision-specific methods discussed in the related work, such as ViF with attention reallocation and L²-VMAS with learned latent memories, would help establish how V-Guide compares with alternative approaches designed for visual collaboration.

Questions for authors
  1. KV-cache aggregation and positional encoding in Hierarchical MAS. Figure 2(c) shows that the latent memories of parallel specialists are merged before being passed to the final answering agent. Section 3.2 and Appendix A explain how visual token positions are aligned across agents. How are the cached keys and values from independent branches combined when their text and latent sequences occupy overlapping positions after the shared visual prefix? Please specify the cache assembly order, whether the original RoPE positions are retained, reassigned to contiguous positions, or otherwise transformed, and how the answering agent’s starting position is determined. These details are needed to reproduce the hierarchical architecture and understand how downstream attention uses the merged cache.

  2. Reliability of prefix-derived visual attention during latent reasoning. What motivates computing visual attention from textual-prefix queries, and how well do those attention distributions reflect the visual regions used during subsequent latent reasoning? Does the consensus include the final answering agent’s attention distribution? If so, how is that distribution computed before the shared visual values are modulated, and are the answering agent’s prefix keys and values reused or recomputed during final answer generation?

2 comments
AuthorsOpenReview ↗

Point-to-Point Response to Reviewer G1Rz [1/2]

Glad that our principled visual consensus design and strong accuracy–efficiency gains resonated with you! Thank you for your insightful review. We respond below.

W1&Q1: KV-Cache Aggregation and Positional Encoding in Hierarchical MAS.

A1: Thanks for your careful review! We clarify the implementation details below.

  • KV-cache aggregation. Three specialist agents independently perform latent reasoning using the shared visual context and their respective role-specific prefixes. The shared visual KV cache is retained only once, while the agent-specific textual-prefix and latent reasoning KV states are concatenated along the sequence dimension in a fixed order.

  • RoPE positional alignment. We re-encode their cached Keys to non-overlapping, contiguous positions in the aggregated sequence. Specifically, we remove the original RoPE rotations and apply those corresponding to the reassigned positions, following the backbone's positional encoding scheme. The Values remain unchanged, as RoPE is applied only to Queries and Keys. The final answering agent then attends to the aggregated cache using subsequent position indices.

We clarify these implementation details in Appendix A to improve reproducibility.

W2: Homogeneous Backbone Constraint and Shared Perceptual Biases.

A2: We appreciate the thoughtful question! We emphasize that homogeneous MAS provides a meaningful and practical setting for training-free latent visual collaboration and further clarify that shared perceptual errors are not unique to homogeneous MAS and examine V-Guide's performance in such challenging cases.

(1) Homogeneous Backbones Enable Efficient, Training-Free Latent Collaboration. We adopt homogeneous agents to enable direct KV-cache sharing and latent collaboration without additional training or cross-model representation alignment. By reusing shared visual KV caches and communicating through latent reasoning kv states, V-Guide achieves 1.7×–4.4× inference speedups and 32%–66% lower token costs compared with Text-VMAS (Fig. 3), demonstrating the practical value of this setting. While heterogeneous agents may offer greater model diversity, directly sharing KV states across different backbones requires addressing incompatible representation spaces, cache structures, and visual tokenization. Such cross-model alignment constitutes a distinct research challenge, which we leave for future work.

(2) Homogeneous Agents Can Provide Complementary Visual Evidence and Informative Consensus. Sharing the same backbone does not imply identical visual attention across agents. As shown in Eq. 1, agents share visual Keys but produce distinct attention distributions through role- and context-conditioned Queries. Our analyses provide three supporting findings:

  • Complementary Visual Coverage. On VStar, the answering agent's top-3 attended regions cover 69.11% of GT targets, increasing to 82.20% when considering all four agents. This demonstrates that other agents can identify relevant evidence missed by the answering agent.
  • Informative Cross-Agent Agreement. Visual tokens jointly prioritized by all four agents achieve a 4.86% GT-region hit rate, compared with 0.08% for unsupported tokens (Fig. 1(a), Appendix C), indicating that cross-agent agreement is informative of task-relevant evidence.
  • Predictive Benefits of Consensus. Under identical modulation strength, cross-agent consensus outperforms final-agent attention guidance by 3.53 points on RealWorldQA (Table 4), demonstrating the benefit of incorporating multi-agent visual attention.

(3) Shared Off-Target Attention Is Not Unique to Homogeneous MAS. Multiple agents may focus on similar misleading regions, but this challenge is not exclusive to homogeneous MAS. Even VLMs from different model families may also be susceptible to similar visual distractors [a,b].

To examine this issue, we analyze 191 all VStar samples with GT bounding-box annotations. We identify 7 cases where the intersection of all four agents' top-3 attended regions is nonempty but does not overlap the GT target. In these cases, V-Guide corrects 4 errors made by Latent-VMAS without introducing any new errors.

Although this subset is small, the results suggest that V-Guide can remain effective despite shared off-target attention. One possible explanation is that its soft consensus incorporates the full attention distributions, while bounded Value modulation preserves access to lower-consensus evidence rather than discarding it.

Overall, homogeneous agents can provide complementary visual evidence and informative consensus, and V-Guide can improve predictions even in the observed shared off-target cases. Further details are provided in Appendix C.6.

[a] Are Reasoning Vision-Language Models Robust to Semantic Visual Distractions?

[b] The Visual Insensitivity Gap: Diagnosing When Vision-Language Models Fail to Use Visual Evidence

AuthorsOpenReview ↗

Point-to-Point Response to Reviewer G1Rz [2/2]

W3: Comparisons with Vision-Specific Multi-Agent Methods.

A3: Valuable suggestion! Direct comparisons with training-based ViF and L2-VMAS are challenging due to reproducibility limitations and differences in experimental settings. Specifically, ViF's released code lacks a complete real-VLM evaluation pipeline, while L2-VMAS has no publicly accessible implementation. Moreover, differences in agent configurations, collaboration mechanisms, and evaluation protocols make direct comparisons of published results difficult to interpret fairly.

To strengthen our evaluation beyond general-purpose multi-agent protocols, we additionally evaluate DART (EACL 2026) [c] under the same Qwen3-VL-8B backbone and benchmark evaluation protocol, while preserving its tool-based collaboration mechanism. We configure four primary agents for the comparison, with DART's method-specific tool recruitment and aggregation components retained.

MethodMMStarRealWorldQA
DART70.4072.94
V-Guide (Ours)72.0773.86

V-Guide outperforms DART on both benchmarks without relying on external visual tools, providing additional evidence of its effectiveness against a vision-specific multi-agent framework. We have incorporated these results into Section 4.

[c] DART: Leveraging Multi-Agent Disagreement for Tool Recruitment in Multimodal Reasoning

Q2: Reliability of Prefix-Derived Visual Attention and Answering-Agent Cache Handling.

A4: Good suggestion! We clarify our design choices and implementation details, supported by additional ablation experiments.

1. Prefix vs. Latent Query States. We adopt textual-prefix Query states as a simple readout of role-conditioned visual attention, avoiding the aggregation of Query states across multiple latent reasoning steps. To examine whether incorporating latent reasoning states provides additional benefits, we conduct the following ablations:

Query SourceVStarRealWorldQA
Prefix Query (Original)85.3473.86
Latent Query85.8673.46
Prefix + Latent Query85.3473.59

As shown above, all three Query sources achieve comparable performance on both benchmarks, indicating that effective visual consensus can be constructed from either prefix or latent reasoning states. This supports our choice of prefix-derived attention as a simpler alternative.

2. Answering-Agent Attention and KV Cache Handling. Our consensus includes all four agents. The final answering agent's visual attention is extracted during its prefix forward pass, before Value modulation and answer generation, thereby avoiding circular dependency. The resulting prefix KV states are retained, while the shared visual Values are subsequently modulated for autoregressive decoding. This avoids redundant prefix computation while allowing subsequent decoding steps to leverage consensus-modulated visual information.

We have clarified the answering-agent attention and KV-cache handling in Appendix A, and incorporated the Query-source ablation results into Section 4.4 (Table 7) of the revised manuscript.


Flag For Ethics Review: Yes

Thank you for highlighting the importance of ethical considerations. V-Guide is a training-free framework evaluated on existing visual reasoning benchmarks. We recognize that real-world applications may involve privacy-sensitive visual data and that cross-agent consensus does not eliminate potential model biases or incorrect visual grounding. To clarify these considerations, we have added a dedicated Ethics Statement to the revised manuscript, discussing potential risks and responsible deployment.


We appreciate again your thoughtful review and hope we addressed all your concerns. Please let us know if you'd like any further information.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.