acceptodds
Under review as a conference paper at ICLR 2027

Draft-KV: Learning Useful Latent Communication Between Language Models

23 views

Abstract

Latent communication passes internal states between language models instead of decoded text, but higher receiver accuracy does not show that the receiver used the message content. Across five method–dataset pairs, replacing each message with one from an unrelated question changes accuracy by at most points, even when communication adds points over the receiver alone. Thus the interface can supply the gain while making the sharer dispensable. instead sends the key–value states formed while the sharer drafts an answer to the current question. Linear projections place these states in a side memory read through a gated attention branch, and progressive training moves from message reconstruction to answer supervision under a guard on harm from mismatched messages. Both models remain frozen and the interface trains M parameters, fewer than C2C. With a Qwen3-8B sharer, a frozen Qwen2.5-0.5B-Instruct receiver reaches on MMLU-Redux, versus alone and with reassigned messages. At fixed interface size, scaling the sharer from B to B raises accuracy from to ; communication also transfers to held-out tasks and can exceed both models when each holds different evidence.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Agents Communication, LLM Collaboration

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_fHTMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies latent communication between language models and argues that improvements in receiver accuracy are insufficient evidence that the receiver actually uses the information supplied by the sharer. The paper introduces a useful diagnostic, pairing gain, defined as the performance difference between receiving the message generated for the current example and receiving a message generated for an unrelated example. Using this intervention, the authors find that existing latent-communication approaches such as C2C and LatentMAS can exhibit substantial system-level gains while having near-zero pairing gains, suggesting that much of their improvement can arise from the trained interface rather than instance-specific information from the sharer.

To address this issue, the paper proposes Draft-KV. A frozen sharer first generates a textual draft, after which the key/value states corresponding to the draft positions are extracted and linearly projected into a side memory accessible through gated attention branches in a frozen receiver. The interface is trained progressively through message reconstruction, answer alignment, and downstream answer supervision with a guard against damage from mismatched messages.

I find the central diagnostic insightful and the empirical phenomenon important. However, I have several concerns about the novelty positioning and the strength of some of the causal claims.

Critical strengths
  1. The pairing-gain diagnostic is useful and addresses a real evaluation problem. The distinction between overall system gain and gain attributable to the correctly paired message is conceptually simple but important. In particular, the C2C analysis demonstrates why an increase in end-task accuracy does not by itself establish successful communication.
Critical weaknesses
  1. What attributes the the performance gain is unclear. The paper does not include perhaps the most important controls: fine-tuning/adapting the sharer or receiver using the same downstream training data and comparable trainable parameter budget. Draft-KV receives substantial task supervision through multiple training stages. It is unclear how much of the improvement should be attributed to “communication” rather than simply adding supervised trainable capacity. Also importantly, the paper should report compute-matched comparisons between baselines. The current results conflate communication representation, additional forward passes, generated reasoning tokens, and total inference compute. The paper itself reports roughly 2× the arithmetic of Sharer-only in the measured setting. Relative to Receiver-only, the inference budget difference is obviously much larger. It is unclear how much of the improvement should be attributed to test-time scaling, i.e., generating more tokens.

  2. The training procedure is highly QA-specific, limiting the claimed generality. Stage 3 is explicitly constructed around question answering: for multiple-choice tasks, the sharer receives the question and candidate options, is instructed to reason and select an option, and the receiver is supervised on the gold answer. Training itself uses ARC-Easy and ARC-Challenge. Consequently, the paper demonstrates a communication method specialized to QA-style answer prediction, rather than a generally applicable protocol for communication between language models. Evaluation is likewise almost entirely multiple-choice or short-answer QA. The paper makes broad claims about “language-model communication,” but provides no evidence on open-ended generation, code, mathematics, planning, dialogue, or multi-step agent collaboration.

Questions for authors
  1. What happens if the receiver or sharer is parameter-efficiently fine-tuned using exactly the same training data?

  2. How does Draft-KV scales at test-time? How does it compare to Receiver-only or Sharer-only under matched test-time computation?

  3. Should the relevant work LMNet [1], which studies trainable dense communication between language models and learns continuous inter-model communication from downstream supervision, be discussed as related work? It appears closely connected to the motivation and background of this paper.

  4. Can the method be demonstrated on a task that is not QA/answer selection?

[1] Language Model Networks: Supervision-Efficient Learning through Dense Communication, ICML 2026/arxiv:2505.12741

7 comments
AuthorsOpenReview ↗

Response [1/3]

Thank you for the detailed review, and for finding the pairing-gain diagnostic insightful and the phenomenon important. We ran the controls you asked for: same-budget fine-tuning of each model, plus our own interface trained without content (Q1, below); compute-matched comparisons with both single models (Q2, comment 2/3); and a new training task, free-form math (W2/Q4, comment 3/3).

[W1] What attributes the performance gain is unclear. The paper does not include perhaps the most important controls: fine-tuning/adapting the sharer or receiver using the same downstream training data and comparable trainable parameter budget. Draft-KV receives substantial task supervision through multiple training stages. It is unclear how much of the improvement should be attributed to "communication" rather than simply adding supervised trainable capacity.

[Q1] What happens if the receiver or sharer is parameter-efficiently fine-tuned using exactly the same training data?

Every control has a trainable budget comparable to our interface's 1,048,584 parameters (exact counts below). On data: Stages 1–2 of Draft-KV use generic OpenHermes conversations rather than benchmark data, so the downstream supervision is ARC for every model below. The LoRA controls train on the same ARC questions and gold answers (3,602 examples, 637 for calibration) for 4,000 updates at batch 16 and lr 10−410^{-4}, keeping the checkpoint with the lowest calibration NLL.

Same pipeline, no content. The control closest to your concern is our own interface, trained through the same three stages, data, update budget, learning rate, batch and seed, with the sharer's message replaced by one fixed neutral passage for every example. It receives all of Draft-KV's supervision and none of the sharer's content.

MMLU-ReduxARC-EasyARC-ChallengeOpenBookQAC-EVAL
Receiver-only37.4563.4740.1043.4041.75
Constant-message interface44.0069.8250.5151.6046.73
Draft-KV (1.7B sharer)65.0792.8983.5376.0061.52

It recovers only 22–25% of the system gain (+4.98 to +10.41 pp), with a pairing gain of exactly zero. Multi-stage supervision and added capacity account for about a quarter of the gain; the rest needs the sharer's message.

Receiver. LoRA on q_proj/v_proj (r=16r=16, 1,081,344 parameters). No message reaches the receiver.

MMLU-ReduxARC-EasyARC-ChallengeOpenBookQAC-EVAL
Receiver-LoRA48.0574.7155.4654.6052.75
Draft-KV (1.7B sharer)65.0792.8983.5376.0061.52
Draft-KV (8B sharer)78.0498.0293.7792.8074.81
Draft-KV gain over Receiver-LoRA (1.7B)+17.02+18.18+28.07+21.40+8.77
Draft-KV gain over Receiver-LoRA (8B)+29.99+23.31+38.31+38.20+22.06

This is the most direct way to spend the same budget and supervision, and it edits attention at every layer, more flexibly than our four-layer read. It still stays 8.8–38.3 pp below Draft-KV. Taken as an upper bound on what supervised capacity can buy, it covers 34–56% of the system gain with the 1.7B sharer and 23–33% with the 8B sharer. The remainder comes from the sharer and grows as the sharer gets stronger, which capacity alone cannot produce. Receiver-LoRA also has to change the receiver's weights; Draft-KV leaves them untouched.

Sharer. LoRA at the same budget (r=5r=5, 1,003,520 parameters at 1.7B; r=2r=2, 958,464 at 8B); the fine-tuned sharer answers on its own.

MMLU-ReduxARC-EasyARC-ChallengeOpenBookQAC-EVAL
Sharer-LoRA, 1.7B61.8892.0079.8676.4060.55
Draft-KV, 1.7B65.0792.8983.5376.0061.52
Sharer-LoRA, 8B75.4397.0191.6486.0075.93
Draft-KV, 8B78.0498.0293.7792.8074.81

Against the same sharers evaluated in the same run, fine-tuning adds 2.8–5.4 pp at 1.7B and at most 1.1 pp at 8B, which is already near its ceiling. Draft-KV beats the fine-tuned sharer in 8 of 10 settings and is within 1.1 pp in the other two, with both of its models frozen.

The same budget and supervision, spent on the receiver, on the sharer, or on our interface without content, does not reproduce the gain. All three controls will go into the paper next to the main results.

AuthorsOpenReview ↗

Response [2/3]

Response to Reviewer fHTM [2/3]

[W1, compute] The current results conflate communication representation, additional forward passes, generated reasoning tokens, and total inference compute. The paper itself reports roughly 2× the arithmetic of Sharer-only in the measured setting. Relative to Receiver-only, the inference budget difference is obviously much larger. It is unclear how much of the improvement should be attributed to test-time scaling.

[Q2] How does Draft-KV scales at test-time? How does it compare to Receiver-only or Sharer-only under matched test-time computation?

Same tokens, nearly the same compute as Sharer-only. Sharer-only, T2T and Draft-KV consume the same sharer draft, so they generate exactly the same tokens; Draft-KV only reads states the sharer has already computed. The 2× figure in our appendix came from a second forward pass we used to extract those states. It is unnecessary: the causal sharer produces the same keys and values while decoding, so we now cache them online. For 8B → 0.5B:

TFLOPs (MMLU-R)× Sharer-onlyMMLU-ReduxARC-Challenge
Sharer-only6.841.00075.0492.83
Text-to-Text7.231.05772.0584.73
Draft-KV, online extraction6.951.01678.0593.26

Online extraction reproduces the main-table predictions on 98.9% and 99.5% of questions and adds about 18 ms per question; we will replace the 2× figure. This separates the factors you list: the generated tokens are identical across the three conditions, none of them runs an extra sharer pass, compute differs by under 6%, and only the way the draft reaches the receiver changes. Draft-KV is ahead of Sharer-only in 19 of 20 sharer settings, and ahead of T2T while costing less.

Against Receiver-only at matched compute. We gave the frozen 0.5B receiver its own test-time compute, chain-of-thought with self-consistency over kk samples, and compare it with Draft-KV under Qwen3 sharers of increasing size (the same 1,000 MMLU-Redux questions; all of ARC-C):

MMLU-R 1k: TFLOPsMMLU-R 1k: acc.ARC-C: TFLOPsARC-C: acc.
Receiver CoT, k=1k=10.3737.500.3240.10
Receiver self-consistency, k=8k=82.9841.002.6046.16
Receiver self-consistency, k=32k=3211.9042.8010.4447.70
Draft-KV, Qwen3-0.6B sharer0.3245.700.2954.52
Draft-KV, Qwen3-1.7B sharer1.4266.301.2483.53
Draft-KV, Qwen3-4B sharer2.9776.702.5291.81
Draft-KV, Qwen3-8B sharer6.9578.605.7693.26

At every budget, spending compute on a sharer beats letting the receiver think longer. At about 0.3 TFLOPs, one receiver CoT pass reaches 40.10 on ARC-C, and Draft-KV with the 0.6B sharer 54.52. At 2.6–3.0 TFLOPs, eight-way self-consistency reaches 41.00 / 46.16, and Draft-KV with the 4B sharer 76.70 / 91.81. Self-consistency also saturates: going from k=8k=8 to k=32k=32 quadruples the cost for 1.5–1.8 pp, and at k=32k=32 it outspends the 8B-sharer system and still trails it by 36–46 pp. The gain comes from the sharer's capability, not from extra tokens.

Scaling. The draft is Draft-KV's only substantial test-time cost, so scaling its compute means scaling the sharer. Along this axis accuracy rises at every step, while the receiver's own curve flattens below 48. We will add this comparison as a figure.

Draft-KV costs 1.016× Sharer-only and less than T2T; at every compute budget, spending it on a sharer beats letting the receiver think longer.

[Q3] Should the relevant work LMNet [1], which studies trainable dense communication between language models and learns continuous inter-model communication from downstream supervision, be discussed as related work?

Thank you, we will add it. LMNet also learns dense communication between language models from end-task supervision, but it builds a trainable network of models whose edges are optimised end to end. Draft-KV connects two frozen models from different families: the message is formed after the sharer drafts, and the receiver reads it as attention memory through about 1M parameters. Our paper also contributes a diagnostic, whether the receiver's gain depends on the paired message, which complements end-to-end trained channels such as LMNet.

AuthorsOpenReview ↗

Response [3/3]

Response to Reviewer fHTM [3/3]

[W2] The training procedure is highly QA-specific, limiting the claimed generality. [...] The paper makes broad claims about "language-model communication," but provides no evidence on open-ended generation, code, mathematics, planning, dialogue, or multi-step agent collaboration.

[Q4] Can the method be demonstrated on a task that is not QA/answer selection?

Draft-KV is a communication architecture: the draft-position KV message, the gated side-memory read, and the three-stage training (reconstruction, alignment, task). None of these is tied to QA. ARC enters only as the training data of the last stage, as any supervised component needs data from its target task. To show this, we changed nothing but that data. From the same 1.7B → 0.5B Stage 2 checkpoint, we ran Stage 3 with the same interface, objective, frozen backbones, sharer drafts (≤512 tokens), update budget (4,000 at batch 16), reconstruction replay and selection rule, on free-form math instead of ARC: 64,000 distinct Orca-Math-200k problems, with the receiver generating "Answer: <number>" and supervised only on the final number, never on worked solutions. We evaluate on GSM8K and SVAMP by exact match on the generated number.

GSM8KSVAMP
Receiver-only48.3753.20
Sharer-only80.3690.40
T2T78.9289.90
Draft-KV (Stage 3 data: Orca-Math)84.1592.80
Draft-KV, deranged45.4149.30

The same method, on generated answers and a different task family, behaves as in the paper. Draft-KV lifts the receiver by 35.8 and 39.6 pp, beats the sharer whose draft it reads by 3.8 and 2.4 pp, and beats T2T on the same draft by 5.2 and 2.9 pp. A deranged draft drops it to just below Receiver-only, so the pairing gain (38.7 and 43.5 pp) accounts for the whole improvement. Stages 1–2 were not rerun: the readability the interface learns once carries over to a new task family. The paper's Private protocol already covers free-form answers: there the receiver decodes a free-form answer and receives the sharer's half of the evidence only through the packet, and Draft-KV beats T2T in all 14 settings (Table 1).

We will add this experiment. The method is the same in every setting; what we will state precisely is the evaluation scope: tasks with a checkable answer, now covering option selection, short-answer QA and free-form math. Long-form generation, code and multi-step agents need training data of their own kind; the architecture and the pairing-gain test apply to them unchanged.

With the method unchanged and only its Stage 3 data swapped, Draft-KV beats the receiver, the sharer and T2T on GSM8K and SVAMP, and its gain again depends entirely on the paired message.

We hope these controls answer W1–W2 and Q1–Q4. We will upload a revised PDF during the discussion period with all of these experiments, the corrected cost figure, the LMNet discussion and the stated scope, and mark every change and addition in blue. If these resolve your concerns, we would be grateful if you would reconsider your score, and we are happy to run anything further during the discussion.

Reviewer_c2aNOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

This paper introduces DraftKV, a latent communications method used for sharing latent embeddings between different models in order to improve performance. The authors examine the effect of message pairing during communication and conclude that it accounts for very little in the improvement. Thus, concluding that the gain must be in the message sharing system rather than the actual shared content. Finally, they introduce DraftKV by using trainable linear mappings between the KV-Caches of models corresponding to the answer being drafted by the sharing model.

Critical strengths

Identifying the information gap, and that the performance gains are perhaps due to fundamentally unintuitive reasons is a significant finding. However, given the critical weaknesses of the paper (mentioned below), this strength is also called into question.

Critical weaknesses

The presentation of the paper is lacking. Very little information is conveyed in the first three sections of the paper. For someone unfamiliar with the surrounding literature, most concepts are not explained clearly, leaving the reader to guess. Section 4 does an acceptable job of explaining the proposed method, however, since the background and supporting concepts are not established, it is not possible to judge the relationship of the method to existing works or its novelty without being already familiar with the literature.

Questions for authors

I don't believe just the answers to some questions could materially affect my assessment, given that I think the writing needs a major overhaul. However, I believe a central problem with the writing is that the core concept, that is, latent communication, and the examined methods that reveal the information-use gap are barely explained. If the authors could elaborate on these concepts, it would be helpful.

4 comments
AuthorsOpenReview ↗

Response to Reviewer c2aN

Thank you for the review, and for noting that the information-use gap is a significant finding. We take the presentation concern seriously. Below we give the explanations you ask for, written as we will add them to the paper, so that you can judge them directly rather than rely on a promise.

I believe a central problem with the writing is that the core concept, that is, latent communication, and the examined methods that reveal the information-use gap are barely explained. If the authors could elaborate on these concepts, it would be helpful.

Latent communication. Multi-LLM systems usually communicate in text: one model decodes its reasoning into tokens, and another reads them back in. Latent communication instead passes internal representations (embeddings, hidden states, or the key–value cache) directly into another model's computation, skipping the decode/re-encode step and keeping information that token sampling discards. The model that sends its states is the sharer, and the model that answers is the receiver. When the two differ in architecture, a small trained interface maps the sharer's states into a form the receiver can attend to.

The methods we examine. C2C (Fu et al., 2025) encodes the question with the sharer, projects its KV cache through a trained fuser, and fuses it into the receiver's cache before the receiver answers. LatentMAS (Zou et al., 2025) lets agents generate latent thoughts and pass their KV memory to the next agent, without training. DLC (Chen et al., 2026) also aligns the sharer's prompt cache to the receiver's. All three report clear accuracy gains for the receiver.

The test, by example. An accuracy gain does not show that the receiver used what the sharer sent, since a trained interface can improve the receiver on its own. Suppose the receiver must answer "Which gas do plants take in for photosynthesis?". Under Matched, it receives the sharer's message for this question. Under Deranged, it receives the sharer's message for another question from the same benchmark, say one about the boiling point of water. Nothing else changes. If the receiver is just as accurate with the wrong message, its gain did not come from what the sharer sent. The pairing gain is this accuracy difference over a benchmark; the system gain is the improvement over the receiver alone.

What we find. For C2C and LatentMAS, the pairing gain is at most 0.60 pp, while the system gain reaches 15.44 pp, and C2C keeps 92% of its gain even when the message is all zeros (new experiment; see our response to Reviewer 2xUf). Draft-KV is built so that the gain depends on the message: the sharer first drafts an answer, and the receiver reads the KV states formed during that drafting. With a Qwen3-8B sharer, the receiver reaches 78.04% on MMLU-Redux with the matched message, 36.40% with a deranged one, and 37.45% alone.

What is new. (i) A criterion, the pairing gain, and with it the finding that existing latent interfaces gain accuracy without using paired content. (ii) A different message: the states formed while the sharer answers, rather than its prompt cache, read as a separate memory through 1.05M parameters (348× fewer than C2C, 36× fewer than DLC). (iii) A result: the gain depends on the paired message in all 35 shared-input settings of Table 1 and grows with sharer capability.

Revisions. The paragraphs above are essentially the text we will add: the first two at the start of the introduction and Section 3, the worked example in Section 3 next to Fig. 2, and a plainer rewrite of the adapter-only paragraph. They add context at a few points and change none of the paper's structure, method, experiments or conclusions, so we see this as a clarification rather than a major overhaul. We will upload the revised PDF during the discussion period, with every change and addition marked in blue.

Your review finds the core result significant and raises no substantive concern about soundness. We sincerely hope that these explanations, and the revised PDF, address your concern about readability. If they do, we would be very grateful if you would consider raising your score. If any passage is still unclear, please point us to it and we will fix it in the revision.

AuthorsOpenReview ↗

Revised PDF uploaded

We have uploaded a revised PDF with the presentation changes described in our response. All changes are marked in blue:

  • Sharer and receiver are defined in the first paragraph of the introduction (Section 1, p. 1).
  • The audited methods are described at the start of Section 3: what each method's message is and where it enters the receiver, for C2C, LatentMAS and DLC (p. 3).
  • A worked example of the Matched/Deranged test is given next to Figure 2 (Section 3, p. 4).
  • The adapter-only evaluation is rewritten in plainer terms (Section 3, Sources of the gain, p. 4).
  • The related work now also covers LMNet, and states how Draft-KV differs (Section 2, p. 2).

None of these changes alters the paper's structure, method, experiments or conclusions. We hope the revised Sections 1–3 read clearly now. If any passage is still hard to follow, please point us to it and we will fix it.

I have read the author's response as well as the blue additions in the revised manuscript. I do believe these changes help with the readability of the paper, and I will thus raise my score by one point (to weak reject).

However, I do still believe the logical thread in the method development is unclear: 1. The source of the information gain gap is still unclear. How does the C2C adapter-only evaluation explain the cause? To me, it's just another example of the information gap 2. I cannot see the connection between the information gain gap and the multi-stage training paradigm introduced in Draft-KV; the same setup could be designed without any knowledge of the information gain gap. What is the connection here? Or the rationale behind the design choices?

It is due to ambiguities like this that are either a problem with the investigation itself or the presentation of the paper that I will still maintain my score below the accept threshold.

AuthorsOpenReview ↗

Response to Reviewer c2aN (follow-up)

Thank you for reading the revision and for raising your score. Both questions go to the logic of the paper, so we answer them directly below. We have also made the same thread explicit in the paper and uploaded a new PDF; changes made in response to this comment are marked in purple (earlier changes remain in blue).

  1. The source of the information gain gap is still unclear. How does the C2C adapter-only evaluation explain the cause?

The two tests answer different questions. The Deranged test shows that C2C's gain does not depend on the paired message. The adapter-only test shows where the gain sits. C2C's fuser takes two inputs, the receiver's own cache and the sharer's cache. When we average its output over messages from five other questions, the receiver still keeps 93–96% of its Matched accuracy (Fig. 1b). When we zero the sharer's cache, C2C still keeps 92% of its gain (Appendix B.4). So the gain is produced by the part of the fuser that depends on the receiver's own cache: the trained interface has become an adapter for the receiver.

We identify two causes, and each one reproduces the gap on its own:

  • What is sent. C2C sends the sharer's prompt cache. Under causal attention these states are computed before the sharer answers, so they hold its reading of the question, which the receiver also has, and not its answer attempt. When we send prompt-position states through our own interface (Prompt KV, Section 5.5), the gap reappears: a 9.73 pp system gain on ARC-C with a 0.17 pp pairing gain.
  • How the interface is trained. An end-to-end task loss rewards any route to the right answer, and nothing in it requires reading the message. Our interface trained on a constant message, with all of Draft-KV's supervision, gains 5–10 pp with a pairing gain of exactly zero (Appendix D.3).
  1. I cannot see the connection between the information gain gap and the multi-stage training paradigm introduced in Draft-KV. [...] Or the rationale behind the design choices?

Each design choice targets one of these causes, and each is judged by the pairing gain rather than by accuracy (pairing gain on ARC-C / MMLU-Redux, Section 5.5 and Appendix D):

CauseDesign choiceEvidence
The message carries no answer attemptSend draft-position states, formed while the sharer answersPP = 45.90 / 28.23, vs. 0.17 / 0.09 with prompt-position states
Task loss can be met without reading the messageStage 1: reconstruction before any task loss. The receiver sees only a fixed instruction and must reproduce text from the packet, so reading it is the only way to lower the lossWithout it, PP drops to 38.74 / 16.58
Stage 2: answer from the sharer's draft states, which may be wrong, so the receiver learns to use them rather than copy themWithout it, PP drops to 41.72 / 19.25
A pairing gain can also be inflated by making wrong messages harmfulStage 3 guard: bounds the loss under a mismatched message at the Receiver-only levelWithout it, PP rises only because Deranged accuracy falls (Appendix D.2)

Could the same setup be designed without knowing the gap? Reconstruction pretraining and curricula exist on their own, and DLC also uses reconstruction. What the gap changes is the target. Judged by accuracy alone, Prompt KV is a working method: it gains 9.73 pp on ARC-C, while C2C's system gain is negative in 29 of 35 settings. Only the pairing gain shows that Prompt KV does not communicate. The diagnosis therefore determined what to send (the draft, which neither C2C nor DLC sends), why readability has to be trained before the task loss, and the criterion that every component had to pass.

Where this now appears in the paper (purple):

  • Section 3, end of Sources of the gain: what the adapter-only test locates, and the two causes with their evidence.
  • Section 4, first sentence (p. 5): the two causes map to Section 4.1 (the message) and Section 4.3 (the training).
  • Section 4.3 (p. 6): the opening, Stage 1 and Stage 3 each state which cause they address.

Thank you for pushing on this; it makes the argument of the paper clearer, and we hope it resolves the remaining ambiguity.

Reviewer_2xUfOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper first asks whether gains from latent communication between LLMs come from the message content or it is artifact of the interface. It define system gain G and pairing gain P, and for C2C and LatentMAS it reports large G but P of at most 0.60 pp. Authors then propose Draft-KV method, where the sharer drafts an answer and the KV states of the draft are sent through a small gated attention branch. The branch is trained in three stages: reconstruction, answer alignment and task training with a one-sided guard. This method shows large P in many tested settings

Critical strengths
  • (S1) The paper is clearly written and well structured; the appendix gives many useful details
  • (S2) Pairing gain is a simple and useful diagnostic, which could be adapted in future works
  • (S3) The C2C audit convincingly shows that the released C2C barely uses the message content
Critical weaknesses
  • (W1) The comparisons C2C vs. receiver-only and Draft-KV vs. T2T are not completely fair, because the first method in each comparison is additionally trained. So we cannot clearly distinguish between gains from training or from gain from the communication channel
  • (W2) Deranged messages come only from the same dataset, and adapter-only donors from the same subject. This cannot separate interface adaptation from dataset-level information such as format or reasoning style. Basically, the C2C gain may come from adapting the receiver or from the general information in dataset, not from the sharer
  • (W3) The paper claims that Draft-KV transfers the sharer’s computation, but the results may have a simpler explanation: the message mostly carries the sharer’s final answer (i.e. the option letter). The Prompt-KV ablation does not separate these two cases, since it only shows that a draft with the answer helps more than a prompt without it
Questions for authors
  • (Q1) What happens to C2C and Draft-KV when the Deranged message comes from another dataset, from a neutral text or from noise?
  • (Q2) Can you train the same interface with a constant input instead of the sharer, and a receiver of similar trainable size that reads the draft as text, using the same data?
  • (Q3) How does Draft-KV perform if only the KV of the final answer line is sent?
  • (Q4) Appendix C.3 gives the sharer prompt for Stage 3 training, but not the prompt used at evaluation. What sharer prompt is used at test time? Is it the same prompt used for Sharer-only?
5 comments
AuthorsOpenReview ↗

Response [1/3]

Thank you for the careful review, and for finding the paper clear, the pairing gain useful, and the C2C audit convincing. We ran every experiment you asked for: Q1 is answered below, Q2 in comment 2/3, and Q3–Q4 in comment 3/3.

[W2] Deranged messages come only from the same dataset, and adapter-only donors from the same subject. This cannot separate interface adaptation from dataset-level information such as format or reasoning style.

[Q1] What happens to C2C and Draft-KV when the Deranged message comes from another dataset, from a neutral text or from noise?

We reran the intervention with four new message sources, keeping the interface checkpoint, receiver prompt and scoring fixed. Entries are the share of the system gain that survives, (Acond−AR)/(AM−AR)(A_\text{cond}-A_R)/(A_M-A_R), in %. Draft-KV uses AR/AMA_R/A_M from Table 1 (37.45/65.07 on MMLU-Redux, 40.10/83.53 on ARC-C); C2C uses the Receiver-only and Matched of the run that produced its interventions, on the pair where its system gain is positive (on 1.7B→0.5B it is negative, so a retention ratio is undefined).

message sourceC2C (0.5B→Q3-0.6B, MMLU)Draft-KV (1.7B→0.5B, MMLU)Draft-KV (1.7B→0.5B, ARC-C)
matched (current question)100100100
deranged, same dataset (paper's protocol)102−2−6
deranged, other dataset97−41−29
neutral text94−6−13
random tokens101−5−6
zero message9200

C2C keeps 92–101% of its gain with random tokens or an all-zero message, which carry neither the sharer's content nor any dataset information. Its gain comes from the trained fuser path itself.

Draft-KV loses the gain under every replacement. The same-dataset row answers W2: that donor keeps the subject, format and answer distribution and changes only the question, and the gain is already gone (−2% / −6%). Dataset-level information is not enough; the gain needs the draft written for this question. Cross-dataset drafts hurt most because they still state an answer, for the wrong question, and the receiver reads it; neutral text and noise state no answer and cost little. A zero message reproduces Receiver-only exactly.

Under every replacement, including no message at all, C2C keeps 92–102% of its gain and Draft-KV keeps none.

The revision will include this table, the donor construction, and donor length statistics.

AuthorsOpenReview ↗

Response [2/3]

[W1] The comparisons C2C vs. receiver-only and Draft-KV vs. T2T are not completely fair, because the first method in each comparison is additionally trained. So we cannot clearly distinguish between gains from training or from gain from the communication channel.

[Q2] Can you train the same interface with a constant input instead of the sharer, and a receiver of similar trainable size that reads the draft as text, using the same data?

One point on W1 first. The pairing gain P=AM−ADP=A_M-A_D is training-matched by construction: Matched and Deranged use the same trained interface on the same questions and differ only in which message arrives, so any effect of training alone cancels. That is why C2C's large GG with near-zero PP is informative, and why Draft-KV's PP (25.65 pp on average over 35 Public settings) cannot be a training effect. The two controls below cover the remaining comparison with T2T. Both are within 3% of our interface's 1,048,584 parameters.

Constant message. Our interface, trained from scratch with the same layers, three stages, data, update budget, learning rate, batch and seed, but with the sharer's message replaced by one fixed 260-token neutral passage (close to the 1.7B sharer's mean draft length). The passage is encoded once and reused for every example, stage and condition, so Matched and Deranged packets are identical.

MMLU-ReduxARC-EasyARC-ChallengeOpenBookQAC-EVAL
Receiver-only37.4563.4740.1043.4041.75
Constant-message interface44.0069.8250.5151.6046.73
Δ\Delta+6.55+6.35+10.41+8.20+4.98

Without sharer content, the same pipeline recovers only about a quarter of the system gain, and its pairing gain is zero. This is the content-independent component that makes up about 92% of C2C's gain, and exactly what PP removes. The revision will report GG and PP side by side for every method, including this control.

Trained text receiver. LoRA (r=16r=16, 1,081,344 parameters) on the receiver's q_proj/v_proj, trained on the same ARC data (3,602 examples, 637 for calibration) with the same supervision: 4,000 updates, batch 16, lr 10−410^{-4}, lowest calibration NLL. At test time it reads the cached sharer draft as text, so both channels receive identical information and differ only in how it is carried.

MMLU-ReduxARC-EasyARC-ChallengeOpenBookQAC-EVAL
T2T, no training (1.7B)60.1988.2675.8565.4053.71
T2T-LoRA (1.7B)65.5791.5881.8375.0060.33
Draft-KV (1.7B)65.0792.8983.5376.0061.52
T2T-LoRA (8B)78.9497.1492.2492.2074.29
Draft-KV (8B)78.0498.0293.7792.8074.81

Training the text channel closes most of the untrained gap (+3.3 to +9.6 pp), so the baseline is not handicapped. Draft-KV still leads by 0.5–1.7 pp on four of five benchmarks at both sharer sizes; MMLU-Redux goes the other way by 0.5 and 0.9 pp, within noise. It also costs less: T2T must prefill the draft through every receiver layer, while Draft-KV reads states the sharer has already computed (1.016× vs. 1.057× Sharer-only FLOPs; see our response to fHTM).

Training alone cancels in PP; trained the same way without content, our interface recovers only a quarter of the gain; and a text receiver trained at the same budget trails Draft-KV in 8 of 10 settings at higher cost.

AuthorsOpenReview ↗

Response [3/3]

[W3] The paper claims that Draft-KV transfers the sharer's computation, but the results may have a simpler explanation: the message mostly carries the sharer's final answer (i.e. the option letter). The Prompt-KV ablation does not separate these two cases.

[Q3] How does Draft-KV perform if only the KV of the final answer line is sent?

We agree that Prompt-KV cannot separate the two. Your Q3 isolates the answer line; we ran it in two versions that expose the same single line and the same choice, and differ only in whether that line carries the sharer's computation:

  • Answer line, with computational states (your Q3). The sharer is prefilled on the full [prompt; draft] as usual, and the receiver's cross-attention is masked to the final answer line. The states at that line are the ones the sharer formed after working through its draft, so they carry its computation.
  • Answer alone, without computational states. The bare sentence "The correct answer is X", with X the sharer's parsed choice, encoded on its own with no prompt or draft and sent through the same interface. This is the "answer only" message of your hypothesis.

Results for 1.7B → 0.5B, on questions whose answer line can be located (MMLU-Redux 5,058 / 5,632; ARC-C 1,161 / 1,172):

MMLU-ReduxARC-C
Receiver-only38.8139.88
Sharer (its parsed answer)62.0480.19
Answer alone, without computational states63.1780.28
Answer line, with computational states68.5184.07
Full Draft-KV (all draft positions)68.7484.07

If the message carried only the answer, Draft-KV would stop at the sharer's accuracy. The answer alone does stop there (63.17 vs. 62.04; 80.28 vs. 80.19). The same line with the sharer's computational states goes 6.5 and 3.9 pp past it. Both name the same choice (the sharer ends its draft with the option text, Appendix C.3), so this margin is what the sharer's computation adds beyond the answer, and no copy of the answer can produce it. Once that line is visible, the rest of the draft adds 0.23 and 0.00 pp. Section 5.2 shows the same over full benchmarks: on ARC-C the receiver recovers 24.1% of the questions the sharer gets wrong while losing 1.1% of those it gets right.

This sharpens our claim: the message does carry the sharer's computation, condensed into the states where it commits to an answer, and the receiver reads more there than the answer itself. We will state the claim this way and add both controls to the ablations. A practical consequence is that the packet can be cut to its last line with no measurable loss.

The answer alone brings the receiver to the sharer's accuracy and no further; the same line with the sharer's computational states takes it 3.9–6.5 pp past the sharer.

[Q4] Appendix C.3 gives the sharer prompt for Stage 3 training, but not the prompt used at evaluation. What sharer prompt is used at test time? Is it the same prompt used for Sharer-only?

The sharer uses the Stage 3 prompt of Appendix C.3 at test time as well, with the same 512-token cap. Each draft is generated once and cached: Sharer-only parses its answer from it, T2T passes it as text, and Draft-KV reads its states. All three share the same sharer output, not just the same prompt. We will state this in the evaluation setup.

We hope these results address W1–W3. We will upload a revised PDF during the discussion period with all new controls and the revised claim, and mark every change and addition in blue. If these resolve your concerns, we would be grateful if you would consider raising your score, and we are happy to discuss anything further.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal Summary for Area Chairs

Dear Area Chair, thank you for handling our submission. A brief summary follows.

What the paper shows. Latent communication methods report that a receiver becomes more accurate when it reads another model's states. Our pairing gain asks whether that gain comes from the message: it compares the receiver given its own question's message with another question's. For C2C and LatentMAS it is at most 0.60 pp, even where communication adds 15.44 pp.

Why this matters. A gain that survives any message comes from the interface, not the sharer; C2C keeps 92% of its gain with an all-zero message. Such a channel cannot improve with a stronger partner, and cannot deliver information the receiver lacks. For C2C the second limit is structural: it fuses the sharer's cache of the same input position by position, so both models must read identical context. Draft-KV sends the states the sharer forms while answering from its own context, and delivers both:

  • Stronger partner, larger gain: from a 0.6B to an 8B sharer, mean gain rises 32.6 pp (C2C: 1.4 pp); the receiver beats the sharer in 19 of 20 settings.
  • Context the receiver lacks: with evidence split between the models, Draft-KV beats text-to-text in all 14 settings (+6.41 pp on average).
  • No fragility: a mismatched message costs 1.41 pp under shared input; an empty one reproduces the receiver exactly.

Both models stay frozen; the interface has ~1M parameters, 348× fewer than C2C.

Reviewer status

ReviewerScoreDiscussion
2xUf3All four requested experiments run and added; no further comment
fHTM2Math results "largely address" generality; controls "helpful"; one remaining T2T-LoRA concern, answered with new results (item 2)
c2aN1 → 2Raised score after revision; follow-up questions answered in a further revision

Strengths noted by reviewers

StrengthReviewer's words
A reusable diagnostic for a real evaluation problem"a simple and useful diagnostic, which could be adapted in future works" (2xUf); "addresses a real evaluation problem" (fHTM)
An important finding about existing methods"the C2C audit convincingly shows that the released C2C barely uses the message content" (2xUf); "I find the central diagnostic insightful and the empirical phenomenon important" (fHTM); "a significant finding" (c2aN)
Clear writing and detailed appendix"clearly written and well structured; the appendix gives many useful details" (2xUf)

All three reviewers, including both below the threshold, recognise the central finding, and none raised a substantive soundness concern (c2aN: "no substantive concerns"; 2xUf, fHTM: "minor or fixable"). The two weak rejects rest on presentation, which c2aN found improved, and on the margin over one trained baseline, addressed in item 2. In fHTM's words, "the rebuttal has strengthened the paper."

Concerns addressed. Each concern is an alternative explanation for the gain; we tested each, and none reproduces it.

  1. Added training (2xUf, fHTM). Pairing gain is training-matched by construction. Our interface trained on a constant message recovers 22–25% of the gain with zero pairing gain; a receiver LoRA stays 8.8–38.3 pp below; Draft-KV leads a sharer LoRA in 8 of 10 (App. D.3).
  2. Trained text receiver (2xUf, fHTM). On multiple choice Draft-KV leads T2T-LoRA in 8 of 10 by 0.5–1.7 pp. On GSM8K/SVAMP with four sharers (0.6B–8B), T2T-LoRA gains only 0.1–0.9 pp over T2T and stays at or below the sharer in 7 of 8, while Draft-KV leads it in all 8 (1.9–5.0 pp in seven; 0.4 pp at 8B SVAMP, near 94%). We read the MCQ gap as LoRA mostly fixing option-text-to-letter mapping, which math lacks.
  3. Test-time compute (fHTM). Draft-KV costs 1.016× the sharer, below text-to-text (1.057×); receiver self-consistency at k=32 outspends the 8B system and trails it by 36–46 pp (App. C.5, E.1).
  4. Dataset-level cues (2xUf). With cross-dataset, neutral, random or zero messages, C2C keeps 92–102% of its gain; Draft-KV keeps none (App. B.4).
  5. Copying the answer (2xUf). The answer alone reaches the sharer's accuracy; the answer line with the sharer's states goes 3.9–6.5 pp past it (App. D.4).
  6. Beyond QA (fHTM). Swapping only the final-stage data for math, Draft-KV beats receiver, sharer and text-to-text (App. D.5).
  7. Presentation and logic (c2aN). Added definitions and a worked example; mapped the gap's two causes to design choices with evidence (Sec. 3–4).
  8. Related work (fHTM). LMNet added (Sec. 2).

Changes are marked in blue (purple for c2aN's follow-up); the four-scale math table will be added to App. D.5. Thank you for your time.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.