acceptodds
Under review as a conference paper at ICLR 2027

MMA: Efficient Latent Communication for Parallel Large Language Model Agents

Abstract

Large Language Models (LLMs) process context through a sequential token interface. However, current multi-agent systems naturally involve parallel branches with disjoint evidence or private contexts. Existing agent communication methods through text or sequence-valued latent space incur decoding re-prefill overhead and introduce communication costs that grow with context length. In this paper, we propose Multi-Modal Attention (MMA), a latent communication framework for frozen parallel LLMs. Each branch retains its context and Key Value (KV) cache locally, exchanging only current-token hidden states at sparse synchronization layers. Furthermore, a latent-fusion aggregator produces a shared next-token distribution for all branches. Because MMA communicates current-token states rather than sequence-length-dependent memories, its payload is independent of input length, and the same interface naturally extends to multiple branches, heterogeneous model widths, and different input modalities. Evaluated on information retrieval benchmarks spanning different context lengths, MMA matches or outperforms other communication baselines while remaining efficient with significantly less communication payload. Its performance advantages are most pronounced in long-context, multi-branch retrieval. These results support MMA as a scalable communication interface for parallel multi-agent inference.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Model, Agent Communication, Multi-agent System, Latent Communication

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_aWmTOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes an implicit communication framework called "Multi-Modal Attention" (MMA). This framework enables multiple Large Language Model (LLM) agents to collaboratively generate a shared response while maintaining their respective local contexts and KV caches. The core concept involves exchanging only the hidden states of the current token at selected Transformer layers, rather than exchanging raw text information or sequence-length-dependent KV caches. MMA introduces a "directional gated cross-branch attention mixer" for intermediate communication and employs a "latent-fusion aggregator" to facilitate shared next-token prediction. Throughout this process, the backbone models remain frozen, with training applied exclusively to the communication and aggregation modules.

Critical strengths
  1. A well-designed, communication-efficient collaborative decoding framework.

This paper addresses a significant limitation in existing "multi-agent communication" methods: the overhead incurred by text-based communication and the growing burden of KV-cache transmission as sequence lengths increase. The proposed framework exchanges only the hidden states at the current timestep, establishing a compact communication interface where the communication load at each step remains independent of the input context length. Furthermore, the shared decoding mechanism eliminates the cumbersome process of generating separate responses for individual agents before aggregating them. This design holds substantial practical value, particularly for distributed, long-context processing scenarios.

  1. Promising experimental results and comprehensive evaluation.

MMA demonstrates strong performance across multiple benchmarks, showing exceptional potential—especially in long-context retrieval and heterogeneous task scenarios. Stress tests regarding context length reveal that, compared to KV-cache-based approaches, this framework significantly reduces the logical communication load while maintaining high accuracy. Ablation studies and "evidence removal" experiments further validate the importance of cross-branch communication. These results provide compelling evidence that the framework effectively integrates information from multiple branches, rather than relying merely on task-specific adaptations.

Critical weaknesses
  1. The core efficiency claims lack validation in a realistic distributed environment.

Although "communication efficiency" is a primary motivation for this work, all latency experiments reported in the paper were conducted on a single NVIDIA RTX 5090 GPU. MMA requires frequent synchronization across multiple Transformer layers during each decoding step; when agents are deployed across different GPUs or machines, this could introduce significant communication and synchronization overhead. Consequently, it remains unclear whether the proposed method truly offers the practical efficiency advantages it claims within the distributed multi-agent scenarios for which it is intended.

  1. The experimental comparisons fail to adequately isolate the specific gains attributable to the proposed communication mechanism.

MMA introduces a substantial number of trainable parameters (approximately 147 million for the Qwen3-1.7B model and 230 million for the Qwen3-4B model), whereas some baseline methods do not require task-specific training on a comparable scale. Furthermore, the architecture is primarily composed of existing components, and the performance improvements over the strongest baselines are relatively modest.

Questions for authors
  1. Distributed Efficiency. Have the authors evaluated the deployment of branched MMA models across different GPUs or machines? Could they provide data on end-to-end latency, throughput, and synchronization overhead within a realistic inter-device communication environment?

  2. Fairness of Baseline Comparisons. To what extent can the performance gains of MMA be attributed specifically to the cyclic communication between branches, rather than to additional trainable parameters or task-specific supervision? Could the authors provide a comparison against a baseline that is more closely matched in terms of model capacity?

Reviewer_cWfBOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

the paper proposes a latent communication framework for multi-agent LLM to collaborate with multi modal latents. Fusion uses a learned aggregator. Experiments demonstrated competitive accuracy vs text based and latent based baselines.

Critical strengths
  1. Seems to be practically implementable for multi modal two-agent communication.

  2. With informative ablation studies on componenets.

Critical weaknesses
  1. Fonts in figures are too small to read.

  2. Very limited novelty compared to related literature. For example, "LLM Augmented LLMs: Expanding Capabilities through Composition" already compose frozen models via learned cross attention, and "Cache-to-Cache: Direct Semantic Communication Between Large Language Models" already studied learned latent transfer.

Questions for authors

N/A

Reviewer_j3u8OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes MMA, a framework in which multiple frozen LLMs process different contexts and exchange current-token hidden states to jointly generate a single response.

Critical strengths

MMA achieves strong task performance while maintaining a small inter-agent communication payload.

Critical weaknesses

MMA’s performance in large-scale serving environments remains insufficiently evaluated. The experiments are conducted on a single GPU, whereas MMA requires cross-branch synchronization at every decoding step. When branches are distributed across multiple GPUs or nodes, frequent synchronization could introduce substantial communication and synchronization overhead despite the small payload. Further evaluation under high request concurrency is also needed.

MMA’s compatibility across different model families remains insufficiently evaluated. Although the experiments include models of different sizes and modalities, all evaluated combinations are within the Qwen family. Experiments involving models from different families are needed to establish the broader applicability of the proposed communication framework.

Questions for authors

1.Could you evaluate MMA’s latency when branches are distributed across multiple GPUs or nodes? Does MMA remain competitive with the baselines when communication and synchronization overhead across devices or nodes is taken into account?

2.Since practical serving systems typically handle multiple concurrent requests, could you compare MMA’s latency and throughput with those of the baselines across different levels of request concurrency?

3.Could you evaluate MMA with combinations of models from different families, such as Qwen and Gemma, to assess whether it remains effective beyond the Qwen-family combinations used in the current experiments?

Comments on the paper

1 comment
AuthorsOpenReview ↗

Open letter to Reviewers cWfB, aWmT, j3u8 and the Area Chair: respond before the discussion closes

Dear Reviewers and Area Chair,

We would like to reminder that the author reviewer discussion ends on Tomorrow. All three ratings are still "2: Weak rejection," and we haven't received responses from the reviewers yet. The Reviewer Guide states: "It is crucial that you are actively engaged during this phase." We look forward to the reviewers' discussion.

Also, we would like to summarize major points for each review report:

Reviewer cWfB. Your review made two critical weakness for rejection: 1. figure fonts (ironically); 2. novelty relative to CALM and C2C and asked no questions. We answered on Nov 6. The fonts are fixed. C2C was already a baseline in the submission. We confirm that MMA beats it on every benchmark with significant less payload and more efficiency. We hope the current version is now acceptable.

Reviewer aWmT. You listed two weaknesses: no real distributed evaluation and no capacity-matched baseline. On Nov 8 we delivered both, a two-machine deployment and a parameter-matched CALM (145M vs. 147M), plus a cross-benchmark transfer study. We are happy that you recognize the framework as "well-designed" and the evaluation "comprehensive." With both weaknesses answered, we hope you agree that now the paper is more acceptable.

Reviewer j3u8. Thank you for engaging on Nov 9 and confirming that the cross-family concern is resolved. We answered both follow-up questions the same day, including the exact concurrent-serving experiment you requested. Your further check is appreciated.

Given that we have successfully addressed the raised concerns, we hope the reviewers could re-evaluate the paper and provide updated comments and decision.

To the Area Chair: We are concerned that we may not have enough time to respond to any remaining concerns, given that reviewers haven't responded to our response yet, while Nov 18 is the deadline. We are wondering whether a reminder message can be sent to the reviewers today (Nov 17).

The Authors of Submission 34781

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.