acceptodds
Under review as a conference paper at ICLR 2027

See What I See, Know What I Think: Dense Latent Communication Across Heterogeneous Agents

Abstract

Language-model agents build internal representations of the information they observe and the reasoning they perform. Sharing these representations offers a way to communicate both source information and reasoning across agents. For agents built from different models, this requires aligning their representations while preserving information useful to the receiver. We study this problem through KV-cache communication, examining how an agent uses internal states shared by other agents, with or without direct access to the information that other agents observed. A controlled self-communication study shows that cache pruning causes substantially greater degradation when the receiving agent lacks access to that information. We use this finding to guide dense cross-model cache alignment, combining positional disentanglement and KV-group transformations with reconstruction followed by generation training. Across six directed Qwen3 pairs, aligned caches improve in-domain accuracy over text communication when both agents observe the same input, with fewer estimated inference FLOPs. Experiments with three-agent document sharing and Mistral-to-Qwen transfer further demonstrate that aligned caches can carry information across both multiple separate observations and different model families.

Primary area:
foundation or frontier models, including LLMs
Keywords:
multi-agent system, latent communication

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_G98bOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a KV-cache alignment framework for latent communication across heterogeneous language-model agents. The method combines positional disentanglement, layer alignment, KV-group transformations, and information selection. It uses a two-stage training method to achieve KV-cache communication and experiments demonstrate promising results across different communication settings.

Critical strengths
  • The paper investigates the distinction between context-aware, context-unaware, and partial-aware communication which provides a useful perspective on heterogeneous agent collaboration.
  • The paper presents an interesting cache-retention study that provides meaningful empirical motivation for the proposed dense alignment framework.
Critical weaknesses
  • The proposed method benefits from task-specific training with gold-answer-guided reasoning traces, while the baselines do not appear to receive comparable supervision. So it may be unclear whether the improvements primarily arise from latent communication or additional task-specific training.
  • The reported FLOPs exclude adapter computation and communication overhead. Given the substantially larger KV-cache payload compared with text messages, actual latency, memory usage, and communication costs should be evaluated.
  • The paper lacks ablations of the two-stage training strategy, KV-group transformations, and learnable gates, making it difficult to identify the contribution of each component.
Questions for authors
  • Can the authors provide more ablations, such as comparing Phase-I-only, Phase-II-only, and the complete two-stage framework?
  • Can the authors report actual inference latency and communication costs to support the claimed efficiency advantages?
3 comments
AuthorsOpenReview ↗

Response to Reviewer G98b

1 Task-specific supervision

Question: Could the gains come from extra task supervision?

Both sender and receiver backbones remain frozen; only the communication adapter is trained. We also evaluate on MMLU-Redux, MedQA, and OpenBookQA, which are excluded from Phase-II training. These results test transfer beyond the supervised tasks, although gains over text communication vary by task and setting.

More directly, our Phase-I-only adapter receives no Phase-II answer supervision yet exceeds the paper’s text baseline on all four evaluated tasks in the context-unaware setting. For Qwen3-8B → Qwen3-4B, GSM8K accuracy is 85.75% vs. 27.98%, and OpenBookQA accuracy is 79.00% vs. 23.20%. Thus, effective communication does not depend solely on Phase-II task supervision. Phase I still uses reconstruction training, so this is not a fully supervision-matched comparison.

2 Training stages and component contributions

Question: What do the two training stages, transformations, and gates contribute?

We completed five ablations on Qwen3-8B → Qwen3-4B, evaluating all four test sets. Phase I uses 30k reconstruction steps on OpenHermes; Phase II uses 2k generation-supervision steps on GSM8K, MATH-algebra, and ARC-C. The single-stage variants omit the other stage. Accuracy is reported in percent; “context-unaware” means the receiver cannot see the question.

Context-aware accuracy

MethodGSM8KMATH-500ARC-COpenBookQA
Full method (paper reference)91.8183.4093.0089.60
Phase I only88.7081.4086.2676.00
Phase II only90.4578.8090.4485.40
Without gates91.1381.0092.2488.20
Without RoPE disentanglement91.3682.2091.4790.20
Shared transformations89.4680.6091.9887.80

Context-unaware accuracy

MethodGSM8KMATH-500ARC-COpenBookQA
Full method (paper reference)91.3681.6093.6090.00
Phase I only85.7575.8084.0479.00
Phase II only19.7123.6072.6159.60
Without gates90.7580.2092.3287.20
Without RoPE disentanglement91.4380.8091.5589.40
Shared transformations78.3273.8089.9382.40

Comparison scope: Full-method scores are reproduced from the paper. New ablations use final checkpoints and separate training artifacts; small differences should not be interpreted as statistically established component effects.

The full method achieves the highest average accuracy across the four benchmarks in both context-aware and context-unaware settings.

3 Actual inference and communication cost

Question: Do latency, memory, and communication measurements support the efficiency claim?

Ours remains 2.37–2.48× faster than text communication, with 50.8–55.0% fewer matrix-operation FLOPs after including adapter transformations and history attention. We measured Mistral-7B → Qwen3-4B on the same 100 examples (25 each from GSM8K, MATH-500, ARC-C, and OpenBookQA), using one H200 and batch size 1. Means per example are:

SettingMethodTime / example ↓Speedup ↑TFLOPs / example ↓FLOPs reduction ↑
Context-awareText14.72 s1.00×14.41—
Context-awareOurs5.94 s2.48×7.0950.8%
Context-unawareText15.52 s1.00×14.19—
Context-unawareOurs6.55 s2.37×6.3855.0%

Latency is CUDA-synchronized wall time after warmup, including tokenization, sender processing, adapter/RoPE operations, and receiver generation. FLOPs are counted separately from executed linear-layer shapes and dense QK/AV attention shapes, including historical KV at each decoding step. Average peak allocated memory is approximately 21.7 vs. 21.2 GiB, and logical payload is 23.56 MiB vs. 563 bytes (ours vs. text); the result supports computational efficiency with reasonable memory usage.

Thanks for clarification and all my concerns have been solved. I will raise my score. Good luck!

Thank you very much for your kind words and support! We truly appreciate your time and thoughtful consideration throughout the review process!

Reviewer_CF4sOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the issues of inconsistent latent representation spaces and incomplete information transmission in heterogeneous LLM multi-agent systems. It proposes a latent space communication method based on Dense KV Cache Alignment. Firstly, the authors conducted KV Group pruning experiments to analyze the impact of different context visibility on information transmission, and found that the receiving end that lacks the original input is more dependent on the complete cache information. On this basis, the method combines position encoding decoupling, cross-layer alignment, mapping per KV Group, and learnable gating, and adopts a two-stage training strategy of KV Cache reconstruction and generation supervision to achieve information transmission between heterogeneous Agents. The authors verified this method in six pairs of Qwen3 models, one cross-model family pairing, and a three-Agent document collaboration task. The experiments show that this method outperforms text communication and existing Cache-to-Cache methods on multiple tasks, while reducing the estimated inference computational cost.

Critical strengths
  1. The method design is reasonable. The author combines position decoupling, KV Group mapping, and two-stage training, while also considering representation reconstruction and downstream task performance, which helps to improve the compatibility of cross-model KV Cache.
  2. The experiments cover various model scales, different context visibility, cross-model family communication and other tasks.
Critical weaknesses
  1. Lack of ablation experiments. The paper introduced Position Disentanglement, KV Group Transformation, Learnable Gates, and two-stage training, but did not provide a complete component ablation. Therefore, it is difficult to clearly identify the independent contributions of each module. In particular, it has not been verified how much performance improvement is brought by Phase I reconstruction pre-training and Phase II generation supervision separately.
  2. Problems with the experimental setup of the paper. In the author's experiments, most of the sender and receiver still used the Qwen3 series models, and the cross-family experiment was only one. It is recommended that the author add more cross-family series experiments to truly verify the effectiveness of this method for communication between heterogeneous agents.
  3. Minor issues. The article's layout is abnormal, such as line 236, 237, 242, 243, etc., which will make the article less readable; the naming is not standardized, for example, in the caption of Figure 4, 'open blue, solid red' the first letters should be capitalized.
Questions for authors
  1. Could the author provide additional ablation experiments to remove Position Disentanglement, Learnable Gates, Phase I Reconstruction, and Phase II Generation Supervision, in order to clarify the specific contributions of each module to the final performance?
  2. The experiments in this paper are mainly based on the Qwen3 series of models, and only include a set of Cross-family model combinations. Could the author further explain whether the current limited cross-model family experiments are sufficient to verify the effectiveness of the method in communication among heterogeneous Agents? Could more communication experiments between different model families be added to further prove the generalization ability and applicability of the method?
  3. Layer Alignment and Group Routing adopt fixed rules. For example, relative depth matching and gmodGSgmod G_S, they do not dynamically find the optimal correspondence relationship based on the actual similarity of cross-model features, but fix the correspondence relationship instead. Will this affect the model performance?
1 comment
AuthorsOpenReview ↗

Response to Reviewer CF4s

1 Component and training-stage ablations

Question: What happens when the proposed components are removed?

We completed five ablations on Qwen3-8B → Qwen3-4B, evaluating all four test sets. Phase I uses 30k reconstruction steps on OpenHermes; Phase II uses 2k generation-supervision steps on GSM8K, MATH-algebra, and ARC-C. The single-stage variants omit the other stage. Accuracy is reported in percent; “context-unaware” means the receiver cannot see the question.

Context-aware accuracy

MethodGSM8KMATH-500ARC-COpenBookQA
Full method (paper reference)91.8183.4093.0089.60
Phase I only88.7081.4086.2676.00
Phase II only90.4578.8090.4485.40
Without gates91.1381.0092.2488.20
Without RoPE disentanglement91.3682.2091.4790.20
Shared transformations89.4680.6091.9887.80

Context-unaware accuracy

MethodGSM8KMATH-500ARC-COpenBookQA
Full method (paper reference)91.3681.6093.6090.00
Phase I only85.7575.8084.0479.00
Phase II only19.7123.6072.6159.60
Without gates90.7580.2092.3287.20
Without RoPE disentanglement91.4380.8091.5589.40
Shared transformations78.3273.8089.9382.40

The full method achieves the highest average accuracy across the four benchmarks in both context-aware and context-unaware settings.

No-gates fixes group weights to one; no-RoPE omits removal/restoration during training and inference; shared mappings use one key MLP and one value MLP across groups, retaining per-group gates.

2 Cross-family evaluation

Question: Can the evidence be extended to additional families?

Transferring rich information between heterogeneous agents remains challenging. We take a step toward this goal by focusing on within-family alignment, complemented by limited cross-family evidence. Our additional experiments in both directions between Qwen and Llama show useful transfer in the context-aware setting: each direction improves over its receiver-only reference on three of four benchmarks. These findings provide a useful starting point for further exploration of cross-family latent communication.

Qwen3-4B → Llama-3.1-8B-Instruct accuracy (%)

MethodGSM8KMATH-500ARC-COpenBookQA
Receiver-only81.1250.0060.4954.80
Ours84.7649.6087.8879.80

Llama-3.1-8B-Instruct → Qwen3-4B accuracy (%)

MethodGSM8KMATH-500ARC-COpenBookQA
Receiver-only (paper)82.4044.2089.2088.00
Ours90.9879.8091.9887.20

3 Fixed layer and group routing

Question: Why use fixed layer and KV-group routing rather than learn the correspondence?

Fixed routing is a deliberate design choice informed by our preliminary experiments. We explored learned routing and observed progressively worse performance as the routing search space expanded. We therefore fix the structural correspondence while retaining learnable group-specific transformations and gates, focusing adaptation on how the selected states are transformed and weighted.

4 Layout and caption consistency

Question: Can the equation spacing and Figure 4 caption capitalization be corrected?

We corrected the equation spacing throughout the manuscript and standardized the color-label capitalization in Figure 4’s caption, with all changes marked in blue.

Reviewer_EMYTOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies dense latent communication across heterogeneous LLM agents through KV-cache alignment. The method combines positional disentanglement, per-group cache transformation, and two-phase training to support both context-aware and context-unaware communication.

Critical strengths
  1. The two-phase training design is easy to understand. Reconstructing receiver-native caches first and then optimizing for downstream generation provides a clear training objective for cross-model latent alignment.
  2. Figure 5 provides a clear conceptual explanation of why dense alignment may be preferable when the receiver does not observe the sender's input, although I think this motivation should appear earlier in the paper.
Critical weaknesses
  1. The Introduction explains what latent communication is, but the limitation of existing latent communication methods is not clearly established in Section 1. The key motivation seems to appear only later: sparse cache transfer works when the receiver already observes the source input, but fails when the cache must carry missing information. This should be introduced much earlier.

  2. Relatedly, Figure 2 mainly explains text vs. latent communication, rather than the proposed method. A method-overview figure around the description in lines 72--73 would be much more useful. Figure 5 also communicates the main sparse-vs-dense motivation well, but appears too late.

  3. Section 3 feels abrupt. The paper introduces compressed sensing and a Lasso surrogate before clearly explaining how this analysis motivates the final method. The logical connection from the retention study to dense reconstruction, group-specific transformations, and generation supervision should be made much clearer.

  4. Cross-family evaluation is limited to only one pair, Mistral-7B to Qwen3-4B. Since heterogeneous communication is a central claim, I think broader cross-family evaluation is important. A compact model (e.g., 4B) from several families, e.g., lower version of Qwen, LLaMA, Gemma, and Mistral, would provide much stronger evidence.

  5. The method contains several important components, including positional disentanglement, layer/group routing, per-group transformations, gates, and two-phase training, but the contribution of each component is not sufficiently isolated. A component-level ablation is needed.

  6. The paper emphasizes lower inference FLOPs, but the reported estimate excludes adapter operations, cache attention cost, serialization, and network transfer. The appendix also reports approximately 27--30 MB payloads for latent communication versus only a few kB for text. I think the efficiency claim should be discussed more carefully from an end-to-end system perspective.

Questions for authors
  1. Does the proposed method require explicit access to the full KV cache at every layer? If so, how compatible is it with FlashAttention or other fused/optimized attention implementations used in practical inference systems?

  2. The method removes the sender RoPE and applies the receiver RoPE after transformation. How do authors ensure that this produces a representation equivalent, or sufficiently close, to the receiver-native positional representation, especially when the sender and receiver use different tokenizers or positional conventions?

  3. Which components contribute most to the final gains: Phase-I reconstruction, Phase-II generation training, positional disentanglement, or the per-group gates?

4 comments
AuthorsOpenReview ↗

Response to Reviewer EMYT - 1

1 Component contributions and two-stage training

Question: What does each component contribute?

We completed five ablations on Qwen3-8B → Qwen3-4B, evaluating all four test sets. Phase I uses 30k reconstruction steps on OpenHermes; Phase II uses 2k generation-supervision steps on GSM8K, MATH-algebra, and ARC-C. The single-stage variants omit the other stage. Accuracy is reported in percent; “context-unaware” means the receiver cannot see the question.

Context-aware accuracy

MethodGSM8KMATH-500ARC-COpenBookQA
Full method (paper reference)91.8183.4093.0089.60
Phase I only88.7081.4086.2676.00
Phase II only90.4578.8090.4485.40
Without gates91.1381.0092.2488.20
Without RoPE disentanglement91.3682.2091.4790.20
Shared transformations89.4680.6091.9887.80

Context-unaware accuracy

MethodGSM8KMATH-500ARC-COpenBookQA
Full method (paper reference)91.3681.6093.6090.00
Phase I only85.7575.8084.0479.00
Phase II only19.7123.6072.6159.60
Without gates90.7580.2092.3287.20
Without RoPE disentanglement91.4380.8091.5589.40
Shared transformations78.3273.8089.9382.40

Comparison scope: Full-method scores are reproduced from the paper. New ablations use final checkpoints and separate training artifacts; small differences should not be interpreted as statistically established component effects.

The full method achieves the highest average accuracy across the four benchmarks in both context-aware and context-unaware settings.

2 Cross-family evidence

Question: Is transfer supported beyond one cross-family pair?

Transferring rich information between heterogeneous agents remains challenging. We take a step toward this goal by focusing on within-family alignment, complemented by limited cross-family evidence. Our additional experiments in both directions between Qwen and Llama show useful transfer in the context-aware setting: each direction improves over its receiver-only reference on three of four benchmarks. These findings provide a useful starting point for further exploration of cross-family latent communication.

Qwen3-4B → Llama-3.1-8B-Instruct accuracy (%)

MethodGSM8KMATH-500ARC-COpenBookQA
Receiver-only81.1250.0060.4954.80
Ours84.7649.6087.8879.80

Llama-3.1-8B-Instruct → Qwen3-4B accuracy (%)

MethodGSM8KMATH-500ARC-COpenBookQA
Receiver-only (paper)82.4044.2089.2088.00
Ours90.9879.8091.9887.20

3 Computation and communication cost

Question: What changes when adapter operations, history attention, and communication are counted?

Ours remains 2.37–2.48× faster than text communication, with 50.8–55.0% fewer matrix-operation FLOPs after including adapter transformations and history attention. We measured Mistral-7B → Qwen3-4B on the same 100 examples (25 each from GSM8K, MATH-500, ARC-C, and OpenBookQA), using one H200 and batch size 1. Means per example are:

SettingMethodTime / example ↓Speedup ↑TFLOPs / example ↓FLOPs reduction ↑
Context-awareText14.72 s1.00×14.41—
Context-awareOurs5.94 s2.48×7.0950.8%
Context-unawareText15.52 s1.00×14.19—
Context-unawareOurs6.55 s2.37×6.3855.0%

Latency is CUDA-synchronized wall time after warmup, including tokenization, sender processing, adapter/RoPE operations, and receiver generation. FLOPs are counted separately from executed linear-layer shapes and dense QK/AV attention shapes, including historical KV at each decoding step. Average peak allocated memory is approximately 21.7 vs. 21.2 GiB, and logical payload is 23.56 MiB vs. 563 bytes (ours vs. text); the result supports computational efficiency with reasonable memory usage.

4 KV access and optimized attention

Question: Does accessing KV caches prevent compatibility with optimized attention such as FlashAttention?

The method requires layerwise KV access, but does not require eager attention or materializing attention weights, and is not conflict with FlashAttention. We have used FlashAttention kernels for sender prefill and receiver token decoding, and memory-efficient SDPA for receiver prefill with the transferred prefix. Thus, fused attention is used with the injected cache.

AuthorsOpenReview ↗

Response to Reviewer EMYT - 2

5 RoPE and tokenizer alignment

Question: How do we align KV caches when the models use different RoPE encodings and tokenizers?

RoPE differences: We remove the sender’s key rotation, transform the cache with the learned adapter, and apply the receiver’s rotation. RoPE conversion handles explicit positional rotations; the reconstruction-trained adapter learns the cross-model representation mapping. We clarify these distinct roles in Section 4.2.

Tokenizer differences: As described in Section 4.1 and Appendix F, both models read the same text, and reconstruction compares only token positions ending at the same character boundary. These positions correspond to the same text prefix, even when token counts and segmentations differ.

6 Motivation and presentation

Question: Can the motivation and the link between the information-distribution study and alignment design be made clearer?

We have polished the presentation by clarifying the missing-input motivation in the opening paragraph of Section 1 and explaining the retention study’s purpose and Lasso’s role at the start of Section 3.1, with both revisions marked in blue.

Thank you for the response and new experiments. The ablations are convincing and strengthen the paper: Phase II alone works in the context-aware setting but collapses in the context-unaware one, while Phase I retains most of it. I encourage moving this to the main paper. RoPE disentanglement and gates contribute little, though this was tested only on a same-RoPE Qwen pair.

Two concerns remain. First, the latency measurement colocates both models on one GPU, so the ~24 MiB payload (vs. 563 bytes) is never transferred; please qualify the efficiency claim. Second, the new Llama results lack T2T/C2C baselines and the context-unaware setting, so the "heterogeneous" framing should be toned down to match the within-family focus stated in the response.

To be transparent, I still view this paper as borderline because of these remaining concerns. However, the authors engaged thoroughly with my comments and resolved my main soundness concerns, so I will raise my score to 3.

Thank you for the thoughtful follow-up and for recognizing that the additional experiments address your main soundness concerns. Following your suggestion, we have added a compact ablation table and discussion to Section 5.4 of the main paper, marked in purple.

Question: Does the latency measurement account for cache transfer?

Thank you for pointing this out. The reported 2.37–2.48× speedup measures colocated inference on one H200 and excludes inter-device transfer. We additionally measured actual KV-cache transfers between two NVLink-connected H200 GPUs: across 20 samples averaging 24.81 MiB, transfer took 0.87 ms on average (400 timed copies). Thus, transfer overhead is small on this interconnect.

Question: Should the heterogeneous framing be narrowed to match the evidence?

Thank you for this suggestion. We have further clarified the challenge, primary within-family scope, and complementary cross-family setting in the introduction and limitations, with the revisions marked in purple.

Thanks again for your thoughtful feedback. Please let us know if you have any remaining concerns!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.