- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies latent communication between language models and argues that improvements in receiver accuracy are insufficient evidence that the receiver actually uses the information supplied by the sharer. The paper introduces a useful diagnostic, pairing gain, defined as the performance difference between receiving the message generated for the current example and receiving a message generated for an unrelated example. Using this intervention, the authors find that existing latent-communication approaches such as C2C and LatentMAS can exhibit substantial system-level gains while having near-zero pairing gains, suggesting that much of their improvement can arise from the trained interface rather than instance-specific information from the sharer.
To address this issue, the paper proposes Draft-KV. A frozen sharer first generates a textual draft, after which the key/value states corresponding to the draft positions are extracted and linearly projected into a side memory accessible through gated attention branches in a frozen receiver. The interface is trained progressively through message reconstruction, answer alignment, and downstream answer supervision with a guard against damage from mismatched messages.
I find the central diagnostic insightful and the empirical phenomenon important. However, I have several concerns about the novelty positioning and the strength of some of the causal claims.
Critical strengths
- The pairing-gain diagnostic is useful and addresses a real evaluation problem. The distinction between overall system gain and gain attributable to the correctly paired message is conceptually simple but important. In particular, the C2C analysis demonstrates why an increase in end-task accuracy does not by itself establish successful communication.
Critical weaknesses
-
What attributes the the performance gain is unclear. The paper does not include perhaps the most important controls: fine-tuning/adapting the sharer or receiver using the same downstream training data and comparable trainable parameter budget. Draft-KV receives substantial task supervision through multiple training stages. It is unclear how much of the improvement should be attributed to “communication” rather than simply adding supervised trainable capacity. Also importantly, the paper should report compute-matched comparisons between baselines. The current results conflate communication representation, additional forward passes, generated reasoning tokens, and total inference compute. The paper itself reports roughly 2× the arithmetic of Sharer-only in the measured setting. Relative to Receiver-only, the inference budget difference is obviously much larger. It is unclear how much of the improvement should be attributed to test-time scaling, i.e., generating more tokens.
-
The training procedure is highly QA-specific, limiting the claimed generality. Stage 3 is explicitly constructed around question answering: for multiple-choice tasks, the sharer receives the question and candidate options, is instructed to reason and select an option, and the receiver is supervised on the gold answer. Training itself uses ARC-Easy and ARC-Challenge. Consequently, the paper demonstrates a communication method specialized to QA-style answer prediction, rather than a generally applicable protocol for communication between language models. Evaluation is likewise almost entirely multiple-choice or short-answer QA. The paper makes broad claims about “language-model communication,” but provides no evidence on open-ended generation, code, mathematics, planning, dialogue, or multi-step agent collaboration.
Questions for authors
-
What happens if the receiver or sharer is parameter-efficiently fine-tuned using exactly the same training data?
-
How does Draft-KV scales at test-time? How does it compare to Receiver-only or Sharer-only under matched test-time computation?
-
Should the relevant work LMNet [1], which studies trainable dense communication between language models and learns continuous inter-model communication from downstream supervision, be discussed as related work? It appears closely connected to the motivation and background of this paper.
-
Can the method be demonstrated on a task that is not QA/answer selection?
[1] Language Model Networks: Supervision-Efficient Learning through Dense Communication, ICML 2026/arxiv:2505.12741
7 comments
Response [2/3]
Response to Reviewer fHTM [2/3]
[W1, compute] The current results conflate communication representation, additional forward passes, generated reasoning tokens, and total inference compute. The paper itself reports roughly 2× the arithmetic of Sharer-only in the measured setting. Relative to Receiver-only, the inference budget difference is obviously much larger. It is unclear how much of the improvement should be attributed to test-time scaling.
[Q2] How does Draft-KV scales at test-time? How does it compare to Receiver-only or Sharer-only under matched test-time computation?
Same tokens, nearly the same compute as Sharer-only. Sharer-only, T2T and Draft-KV consume the same sharer draft, so they generate exactly the same tokens; Draft-KV only reads states the sharer has already computed. The 2× figure in our appendix came from a second forward pass we used to extract those states. It is unnecessary: the causal sharer produces the same keys and values while decoding, so we now cache them online. For 8B → 0.5B:
| TFLOPs (MMLU-R) | × Sharer-only | MMLU-Redux | ARC-Challenge | |
|---|---|---|---|---|
| Sharer-only | 6.84 | 1.000 | 75.04 | 92.83 |
| Text-to-Text | 7.23 | 1.057 | 72.05 | 84.73 |
| Draft-KV, online extraction | 6.95 | 1.016 | 78.05 | 93.26 |
Online extraction reproduces the main-table predictions on 98.9% and 99.5% of questions and adds about 18 ms per question; we will replace the 2× figure. This separates the factors you list: the generated tokens are identical across the three conditions, none of them runs an extra sharer pass, compute differs by under 6%, and only the way the draft reaches the receiver changes. Draft-KV is ahead of Sharer-only in 19 of 20 sharer settings, and ahead of T2T while costing less.
Against Receiver-only at matched compute. We gave the frozen 0.5B receiver its own test-time compute, chain-of-thought with self-consistency over samples, and compare it with Draft-KV under Qwen3 sharers of increasing size (the same 1,000 MMLU-Redux questions; all of ARC-C):
| MMLU-R 1k: TFLOPs | MMLU-R 1k: acc. | ARC-C: TFLOPs | ARC-C: acc. | |
|---|---|---|---|---|
| Receiver CoT, | 0.37 | 37.50 | 0.32 | 40.10 |
| Receiver self-consistency, | 2.98 | 41.00 | 2.60 | 46.16 |
| Receiver self-consistency, | 11.90 | 42.80 | 10.44 | 47.70 |
| Draft-KV, Qwen3-0.6B sharer | 0.32 | 45.70 | 0.29 | 54.52 |
| Draft-KV, Qwen3-1.7B sharer | 1.42 | 66.30 | 1.24 | 83.53 |
| Draft-KV, Qwen3-4B sharer | 2.97 | 76.70 | 2.52 | 91.81 |
| Draft-KV, Qwen3-8B sharer | 6.95 | 78.60 | 5.76 | 93.26 |
At every budget, spending compute on a sharer beats letting the receiver think longer. At about 0.3 TFLOPs, one receiver CoT pass reaches 40.10 on ARC-C, and Draft-KV with the 0.6B sharer 54.52. At 2.6–3.0 TFLOPs, eight-way self-consistency reaches 41.00 / 46.16, and Draft-KV with the 4B sharer 76.70 / 91.81. Self-consistency also saturates: going from to quadruples the cost for 1.5–1.8 pp, and at it outspends the 8B-sharer system and still trails it by 36–46 pp. The gain comes from the sharer's capability, not from extra tokens.
Scaling. The draft is Draft-KV's only substantial test-time cost, so scaling its compute means scaling the sharer. Along this axis accuracy rises at every step, while the receiver's own curve flattens below 48. We will add this comparison as a figure.
Draft-KV costs 1.016× Sharer-only and less than T2T; at every compute budget, spending it on a sharer beats letting the receiver think longer.
[Q3] Should the relevant work LMNet [1], which studies trainable dense communication between language models and learns continuous inter-model communication from downstream supervision, be discussed as related work?
Thank you, we will add it. LMNet also learns dense communication between language models from end-task supervision, but it builds a trainable network of models whose edges are optimised end to end. Draft-KV connects two frozen models from different families: the message is formed after the sharer drafts, and the receiver reads it as attention memory through about 1M parameters. Our paper also contributes a diagnostic, whether the receiver's gain depends on the paired message, which complements end-to-end trained channels such as LMNet.
Response [3/3]
Response to Reviewer fHTM [3/3]
[W2] The training procedure is highly QA-specific, limiting the claimed generality. [...] The paper makes broad claims about "language-model communication," but provides no evidence on open-ended generation, code, mathematics, planning, dialogue, or multi-step agent collaboration.
[Q4] Can the method be demonstrated on a task that is not QA/answer selection?
Draft-KV is a communication architecture: the draft-position KV message, the gated side-memory read, and the three-stage training (reconstruction, alignment, task). None of these is tied to QA. ARC enters only as the training data of the last stage, as any supervised component needs data from its target task. To show this, we changed nothing but that data. From the same 1.7B → 0.5B Stage 2 checkpoint, we ran Stage 3 with the same interface, objective, frozen backbones, sharer drafts (≤512 tokens), update budget (4,000 at batch 16), reconstruction replay and selection rule, on free-form math instead of ARC: 64,000 distinct Orca-Math-200k problems, with the receiver generating "Answer: <number>" and supervised only on the final number, never on worked solutions. We evaluate on GSM8K and SVAMP by exact match on the generated number.
| GSM8K | SVAMP | |
|---|---|---|
| Receiver-only | 48.37 | 53.20 |
| Sharer-only | 80.36 | 90.40 |
| T2T | 78.92 | 89.90 |
| Draft-KV (Stage 3 data: Orca-Math) | 84.15 | 92.80 |
| Draft-KV, deranged | 45.41 | 49.30 |
The same method, on generated answers and a different task family, behaves as in the paper. Draft-KV lifts the receiver by 35.8 and 39.6 pp, beats the sharer whose draft it reads by 3.8 and 2.4 pp, and beats T2T on the same draft by 5.2 and 2.9 pp. A deranged draft drops it to just below Receiver-only, so the pairing gain (38.7 and 43.5 pp) accounts for the whole improvement. Stages 1–2 were not rerun: the readability the interface learns once carries over to a new task family. The paper's Private protocol already covers free-form answers: there the receiver decodes a free-form answer and receives the sharer's half of the evidence only through the packet, and Draft-KV beats T2T in all 14 settings (Table 1).
We will add this experiment. The method is the same in every setting; what we will state precisely is the evaluation scope: tasks with a checkable answer, now covering option selection, short-answer QA and free-form math. Long-form generation, code and multi-step agents need training data of their own kind; the architecture and the pairing-gain test apply to them unchanged.
With the method unchanged and only its Stage 3 data swapped, Draft-KV beats the receiver, the sharer and T2T on GSM8K and SVAMP, and its gain again depends entirely on the paired message.
We hope these controls answer W1–W2 and Q1–Q4. We will upload a revised PDF during the discussion period with all of these experiments, the corrected cost figure, the LMNet discussion and the stated scope, and mark every change and addition in blue. If these resolve your concerns, we would be grateful if you would reconsider your score, and we are happy to run anything further during the discussion.
Response [1/3]
Thank you for the detailed review, and for finding the pairing-gain diagnostic insightful and the phenomenon important. We ran the controls you asked for: same-budget fine-tuning of each model, plus our own interface trained without content (Q1, below); compute-matched comparisons with both single models (Q2, comment 2/3); and a new training task, free-form math (W2/Q4, comment 3/3).
Every control has a trainable budget comparable to our interface's 1,048,584 parameters (exact counts below). On data: Stages 1–2 of Draft-KV use generic OpenHermes conversations rather than benchmark data, so the downstream supervision is ARC for every model below. The LoRA controls train on the same ARC questions and gold answers (3,602 examples, 637 for calibration) for 4,000 updates at batch 16 and lr 10−4, keeping the checkpoint with the lowest calibration NLL.
Same pipeline, no content. The control closest to your concern is our own interface, trained through the same three stages, data, update budget, learning rate, batch and seed, with the sharer's message replaced by one fixed neutral passage for every example. It receives all of Draft-KV's supervision and none of the sharer's content.
It recovers only 22–25% of the system gain (+4.98 to +10.41 pp), with a pairing gain of exactly zero. Multi-stage supervision and added capacity account for about a quarter of the gain; the rest needs the sharer's message.
Receiver. LoRA on
q_proj/v_proj(r=16, 1,081,344 parameters). No message reaches the receiver.This is the most direct way to spend the same budget and supervision, and it edits attention at every layer, more flexibly than our four-layer read. It still stays 8.8–38.3 pp below Draft-KV. Taken as an upper bound on what supervised capacity can buy, it covers 34–56% of the system gain with the 1.7B sharer and 23–33% with the 8B sharer. The remainder comes from the sharer and grows as the sharer gets stronger, which capacity alone cannot produce. Receiver-LoRA also has to change the receiver's weights; Draft-KV leaves them untouched.
Sharer. LoRA at the same budget (r=5, 1,003,520 parameters at 1.7B; r=2, 958,464 at 8B); the fine-tuned sharer answers on its own.
Against the same sharers evaluated in the same run, fine-tuning adds 2.8–5.4 pp at 1.7B and at most 1.1 pp at 8B, which is already near its ceiling. Draft-KV beats the fine-tuned sharer in 8 of 10 settings and is within 1.1 pp in the other two, with both of its models frozen.
The same budget and supervision, spent on the receiver, on the sharer, or on our interface without content, does not reproduce the gain. All three controls will go into the paper next to the main results.