- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates whether the performance gains from cache-to-cache (C2C) communication arise from transferring question-specific information or from correcting the receiver’s output bias. The authors examine these explanations through answer-exclusive transfer tasks, sender-cache ablations, logit-shift decomposition, and comparisons across model sizes and training settings. For the main released 0.6B receiver, replacing the sender’s cache with zeros preserves most of the improvement, while a question-independent logit correction reproduces much of the gain. In contrast, some model pairs retrained on task data exhibit gains that depend on the sender’s content. The paper argues that accuracy improvements alone are insufficient evidence of semantic transfer and proposes controls to distinguish information transfer from output calibration.
Critical strengths
- The paper identifies an important evaluation issue and supports its explanation with complementary experiments. Beyond showing that gains survive zeroing the sender’s cache, the authors provide per-item prediction agreement, logit-shift decomposition, and a calibration baseline requiring neither a sender nor a fuser. Together, these results provide convincing evidence that output-bias correction explains most of the improvement for the main released 0.6B receiver.
- The findings offer practical guidance for evaluating latent communication methods. The main analysis uses released checkpoints, making it directly relevant to the behaviour of existing systems. Zero-cache and output-centring controls are also straightforward to adopt and can help determine whether improvements actually require sender content. By reporting settings where gains do depend on the sender, the paper also helps clarify the scope of the calibration explanation.
Critical weaknesses
Insufficient analysis of when different sources of performance gains emerge. While the paper provides convincing evidence that output calibration explains the gains of the released C2C checkpoints, the conditions under which C2C learns calibration versus sender-dependent information transfer remain less clear. Section 4.4 suggests that receiver selection bias and sender capability play important roles, but these factors are not sufficiently isolated from other differences in model representations and training settings. More controlled experiments that independently vary these factors would help establish a clearer understanding of when and why each type of gain emerges.
Questions for authors
-
Conditions for different gain sources. Section 4.4 suggests that receiver selection bias, sender capability, and training conditions influence whether C2C learns output calibration or sender-dependent transfer. However, these factors are not fully disentangled. Could the authors provide more controlled comparisons to clarify their individual effects?
-
Verdict transfer versus full-content transfer. The 7B 0.6B pair succeeds on colour lookup and transfers the first code digit, but fails to recover the remaining digits. Does this indicate that the remaining information is not preserved in the fused KV representations, or that the receiver cannot effectively extract it? Further analysis of information recoverability could help explain this distinction.
5 comments
The same model pair exhibits different sender dependence under different training conditions (2/2)
Your concern (summary): Receiver bias, sender ability, and training conditions may affect whether C2C learns calibration or sender-dependent behavior, but the original experiments do not sufficiently disentangle them.
1.3 Hold the model pair fixed and compare training data
We compare Qwen2.5-7B → Qwen3-0.6B fusers trained on multiple-choice questions or general dialogue. Both use a 40,000-example cap and one epoch, but length filtering retains 39,867 multiple-choice examples and 38,603 dialogue examples, resulting in 1,234 and 1,195 training steps, respectively:
| Training data | MMLU-Redux: real / zero sender | OpenBookQA: real / zero sender |
|---|---|---|
| Multiple-choice questions | 68.1 / 24.3 | 80.6 / 29.8 |
| General dialogue | 50.0 / 46.1 | 56.8 / 49.6 |
Accuracy (%). Receiver-only accuracy is 34.9% on MMLU-Redux and 39.2% on OpenBookQA.
With multiple-choice training, zeroing causes a large drop; with dialogue training, most of the gain remains. The same model pair therefore exhibits different degrees of sender dependence across these training conditions. Because filtering also changes training exposure, this comparison does not isolate data content from every other training difference.
These results connect the source of improvement to specific conditions rather than assigning every gain to either calibration or information transfer. We will specify what each intervention holds fixed and changes, and limit conclusions to the tested pairings and training settings.
Digit information remains readable from the fused cache despite incomplete answer recovery
Your concern (summary): Qwen2.5-7B → Qwen3-0.6B succeeds on color lookup and the first code digit but not the remaining digits. Is the information lost from the fused KV cache, or retained but inaccessible to the receiver?
Digit information remains linearly readable from the fused cache, but the receiver's predictions for the last three digits stay near chance under the paper's 1,000-step training conditions. Separately, the first digit is much easier to read near the end of the sender's prompt, offering a clue to the asymmetry.
Digit prediction accuracy (%); chance accuracy is 10%.
| Sender | Runs | Digit 1 | Digit 2 | Digit 3 | Digit 4 |
|---|---|---|---|---|---|
| Qwen2.5-0.5B | 3 | 9.3 | 9.8 | 10.4 | 9.6 |
| Qwen3-1.7B | 3 | 98.8 | 99.2 | 99.0 | 99.0 |
| Qwen2.5-3B | 4 | 82.4 | 10.3 | 10.0 | 9.8 |
| Qwen2.5-7B | 1 | 98.9 | 9.8 | 10.2 | 10.7 |
Table R3-2. Teacher-forced per-digit accuracy after 1,000 steps. Receiver: Qwen3-0.6B; learning rate: 10⁻⁴. In the prompt, the sender sees the code and the receiver sees 0000. At each answer position we supply preceding gold digits and choose the highest-scoring token among 0-9. Run counts appear in the Runs column; the four 3B runs cover three seeds, with seed 0 run in two environments. Chance accuracy is 10%.
We probe the fused cache at the prompt positions of the four digits, training on 1,500 questions and testing on 500 separate questions. A separate linear classifier is fitted at each sampled layer; for each digit we report the highest observed test accuracy across those layers, then average over the four digits.
| Measurement for Qwen2.5-7B → Qwen3-0.6B | Accuracy |
|---|---|
| Teacher-forced digit prediction: first digit | Approximately 99% |
| Teacher-forced digit prediction: each remaining digit | Approximately 10% |
| Best-layer linear readout at digit positions in the fused cache | Four-digit mean: approximately 100% |
| Same probe setting, sender cache zeroed | Four-digit mean: approximately 12% |
Table R3-3. Teacher-forced digit prediction versus linear readout after 1,000 fuser-training steps. Probe results are means over all four digits, not only the last three.
These results rule out the simple interpretation that later-digit prediction fails because all corresponding information has disappeared from the fused representation. Moreover, the later digits remain near chance even with gold preceding digits, so accumulated prediction errors alone do not explain the failure. However, an entirely untrained fuser also gives nearly 100% mean readout accuracy. The fusion architecture itself therefore preserves some information, and successful probing does not establish that the trained fuser has learned effective use of it.
We separately probe the last sender-cache position passed to the fuser, near the end of the 7B sender's prompt. Unlike the preceding test, this attempts to read all four digits from the same position, using the same train/test split and best-layer reporting procedure:
| Digit | First | Second | Third | Fourth |
|---|---|---|---|---|
| Best-layer linear-probe accuracy | 99% | 32% | 19% | 17% |
The first digit is much more accessible there, consistent with the prediction results. This suggests a possible route through the end-of-prompt representation, but does not distinguish a representation the receiver cannot readily use from failure to extract retained information at later answer positions.
Color lookup requires one answer letter and may benefit from the same route. This remains a hypothesis: we have not performed the corresponding linear-probe analysis.
We will report per-digit accuracy, exact-match accuracy, and linear recoverability separately, distinguishing information retained in the cache from information the receiver actually uses to answer.
Receiver bias and sender choice affect sender dependence in the tested pairings (1/2)
Thank you for this suggestion. We added experiments that change receiver bias, introduce intermediate sender sizes, and replace the training data. The same pair can exhibit different degrees of sender dependence under different biases or training data; among the tested Qwen2.5 senders, stronger senders also produce fusion predictions that more often follow their answers.
1.1 Hold the model pair fixed and change receiver bias
For Qwen2.5-3B → Qwen3-1.7B, we subtract 20 from the receiver's logits for the B, C, and D answer tokens to favor A. Both language models remain unchanged. We retrain the fuser with this shift and apply the same shift during evaluation, including to the receiver-only baseline.
In the original configuration, sender zeroing reduces fusion accuracy below the receiver alone. With bias injection and retraining, zero-sender improvements appear:
Changing receiver bias and retraining can produce sender-independent gains within the same model pair. We also cross training-time and evaluation-time bias conditions:
OpenBookQA accuracy (%).
Table R3-1. Training and evaluation bias conditions affect fuser performance. Qwen2.5-3B → Qwen3-1.7B on OpenBookQA. Bias means subtracting 20 from the B, C, and D answer-token logits; the baseline has the same evaluation-time shift. This table shows one training run; the second run's zero-sender gains appear in the preceding table. pp denotes percentage points.
With bias present at evaluation, the fuser trained under that bias performs better; without it, that fuser performs worse. Its effect thus depends on the bias encountered during training.
1.2 Hold the receiver fixed and vary the sender
We fix Qwen3-0.6B and add Qwen2.5-1.5B and 3B between the existing 0.5B and 7B senders. Each fuser is retrained using the same multiple-choice training scheme.
Accuracy (%). “Zero sender” denotes zeroing the sender cache before fusion.
Most of the 0.5B gain remains with the sender zeroed, while the other three pairings show stronger sender dependence. On questions where sender and receiver originally disagree, fusion follows the sender on 67%/75% of questions for 3B and 77%/77% for 7B, versus about 27%/26% for 0.5B (MMLU-Redux/OpenBookQA). Together with the cache ablations, this pattern is consistent with greater sender influence for the larger tested senders; prediction agreement alone does not establish causal transfer.
Changing the sender also changes its representations, so this within-series comparison does not completely separate ability from representation differences.