- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes a KV-cache alignment framework for latent communication across heterogeneous language-model agents. The method combines positional disentanglement, layer alignment, KV-group transformations, and information selection. It uses a two-stage training method to achieve KV-cache communication and experiments demonstrate promising results across different communication settings.
Critical strengths
- The paper investigates the distinction between context-aware, context-unaware, and partial-aware communication which provides a useful perspective on heterogeneous agent collaboration.
- The paper presents an interesting cache-retention study that provides meaningful empirical motivation for the proposed dense alignment framework.
Critical weaknesses
- The proposed method benefits from task-specific training with gold-answer-guided reasoning traces, while the baselines do not appear to receive comparable supervision. So it may be unclear whether the improvements primarily arise from latent communication or additional task-specific training.
- The reported FLOPs exclude adapter computation and communication overhead. Given the substantially larger KV-cache payload compared with text messages, actual latency, memory usage, and communication costs should be evaluated.
- The paper lacks ablations of the two-stage training strategy, KV-group transformations, and learnable gates, making it difficult to identify the contribution of each component.
Questions for authors
- Can the authors provide more ablations, such as comparing Phase-I-only, Phase-II-only, and the complete two-stage framework?
- Can the authors report actual inference latency and communication costs to support the claimed efficiency advantages?
3 comments
Thanks for clarification and all my concerns have been solved. I will raise my score. Good luck!
Thank you
Thank you very much for your kind words and support! We truly appreciate your time and thoughtful consideration throughout the review process!
Response to Reviewer G98b
1 Task-specific supervision
Both sender and receiver backbones remain frozen; only the communication adapter is trained. We also evaluate on MMLU-Redux, MedQA, and OpenBookQA, which are excluded from Phase-II training. These results test transfer beyond the supervised tasks, although gains over text communication vary by task and setting.
More directly, our Phase-I-only adapter receives no Phase-II answer supervision yet exceeds the paper’s text baseline on all four evaluated tasks in the context-unaware setting. For Qwen3-8B → Qwen3-4B, GSM8K accuracy is 85.75% vs. 27.98%, and OpenBookQA accuracy is 79.00% vs. 23.20%. Thus, effective communication does not depend solely on Phase-II task supervision. Phase I still uses reconstruction training, so this is not a fully supervision-matched comparison.
2 Training stages and component contributions
We completed five ablations on Qwen3-8B → Qwen3-4B, evaluating all four test sets. Phase I uses 30k reconstruction steps on OpenHermes; Phase II uses 2k generation-supervision steps on GSM8K, MATH-algebra, and ARC-C. The single-stage variants omit the other stage. Accuracy is reported in percent; “context-unaware” means the receiver cannot see the question.
Context-aware accuracy
Context-unaware accuracy
Comparison scope: Full-method scores are reproduced from the paper. New ablations use final checkpoints and separate training artifacts; small differences should not be interpreted as statistically established component effects.
The full method achieves the highest average accuracy across the four benchmarks in both context-aware and context-unaware settings.
3 Actual inference and communication cost
Ours remains 2.37–2.48× faster than text communication, with 50.8–55.0% fewer matrix-operation FLOPs after including adapter transformations and history attention. We measured Mistral-7B → Qwen3-4B on the same 100 examples (25 each from GSM8K, MATH-500, ARC-C, and OpenBookQA), using one H200 and batch size 1. Means per example are:
Latency is CUDA-synchronized wall time after warmup, including tokenization, sender processing, adapter/RoPE operations, and receiver generation. FLOPs are counted separately from executed linear-layer shapes and dense QK/AV attention shapes, including historical KV at each decoding step. Average peak allocated memory is approximately 21.7 vs. 21.2 GiB, and logical payload is 23.56 MiB vs. 563 bytes (ours vs. text); the result supports computational efficiency with reasonable memory usage.