acceptodds
Under review as a conference paper at ICLR 2027

Demystifying Cache-to-Cache Communication: When Latent Transfer Reduces to Output Calibration

Abstract

Recent work proposes cache-to-cache (C2C) communication for heterogeneous LLM agents, directly transferring KV caches to convey question-specific semantics more efficiently than text-based communication. In this paper, we revisit this mechanism and uncover a different explanation: C2C gains arise primarily from a question-independent distributional shift that corrects a degenerate receiver output distribution, rather than from question-specific semantic transfer. Using the authors' released weights and evaluation protocol, we first reproduce the reported gains, then conduct four controlled experiments. First, in a setting where only the sender holds the answer, nothing crosses between the released model pair. Second, replacing the sender KV cache with zeros preserves nearly the full improvement, showing that sender-specific semantic content is unnecessary. Third, decomposing the induced logit shift shows that a single question-independent vector reproduces the gain, while the question-dependent residual is ineffective or harmful. Fourth, varying receiver scale shows that gains appear only when the receiver output distribution is degenerate and disappear after directly centring it. The effect has a boundary: fusers trained on task data do carry the sender's answer when the sender is much stronger or the receiver has no bias to correct. Together, these results challenge the interpretation of C2C as semantic communication and instead attribute its benchmark gains to a content-independent correction of the receiver distribution. Our findings highlight the need to distinguish semantic information transfer from distributional calibration when evaluating latent communication between LLMs. Code, per-item outputs and the data of the constructed tasks are available at https://anonymous.4open.science/r/gains-without-transfer.

Primary area:
foundation or frontier models, including LLMs
Keywords:
cache-to-cache communication, KV cache, latent communication, multi-agent LLMs, label bias

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_jtNgOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether the performance gains from cache-to-cache (C2C) communication arise from transferring question-specific information or from correcting the receiver’s output bias. The authors examine these explanations through answer-exclusive transfer tasks, sender-cache ablations, logit-shift decomposition, and comparisons across model sizes and training settings. For the main released 0.6B receiver, replacing the sender’s cache with zeros preserves most of the improvement, while a question-independent logit correction reproduces much of the gain. In contrast, some model pairs retrained on task data exhibit gains that depend on the sender’s content. The paper argues that accuracy improvements alone are insufficient evidence of semantic transfer and proposes controls to distinguish information transfer from output calibration.

Critical strengths
  1. The paper identifies an important evaluation issue and supports its explanation with complementary experiments. Beyond showing that gains survive zeroing the sender’s cache, the authors provide per-item prediction agreement, logit-shift decomposition, and a calibration baseline requiring neither a sender nor a fuser. Together, these results provide convincing evidence that output-bias correction explains most of the improvement for the main released 0.6B receiver.
  2. The findings offer practical guidance for evaluating latent communication methods. The main analysis uses released checkpoints, making it directly relevant to the behaviour of existing systems. Zero-cache and output-centring controls are also straightforward to adopt and can help determine whether improvements actually require sender content. By reporting settings where gains do depend on the sender, the paper also helps clarify the scope of the calibration explanation.
Critical weaknesses

Insufficient analysis of when different sources of performance gains emerge. While the paper provides convincing evidence that output calibration explains the gains of the released C2C checkpoints, the conditions under which C2C learns calibration versus sender-dependent information transfer remain less clear. Section 4.4 suggests that receiver selection bias and sender capability play important roles, but these factors are not sufficiently isolated from other differences in model representations and training settings. More controlled experiments that independently vary these factors would help establish a clearer understanding of when and why each type of gain emerges.

Questions for authors
  1. Conditions for different gain sources. Section 4.4 suggests that receiver selection bias, sender capability, and training conditions influence whether C2C learns output calibration or sender-dependent transfer. However, these factors are not fully disentangled. Could the authors provide more controlled comparisons to clarify their individual effects?

  2. Verdict transfer versus full-content transfer. The 7B →\rightarrow 0.6B pair succeeds on colour lookup and transfers the first code digit, but fails to recover the remaining digits. Does this indicate that the remaining information is not preserved in the fused KV representations, or that the receiver cannot effectively extract it? Further analysis of information recoverability could help explain this distinction.

5 comments
AuthorsOpenReview ↗

Receiver bias and sender choice affect sender dependence in the tested pairings (1/2)

Your concern (summary): Receiver bias, sender ability, and training conditions may affect whether C2C learns calibration or sender-dependent behavior, but the original experiments do not sufficiently disentangle them.

Thank you for this suggestion. We added experiments that change receiver bias, introduce intermediate sender sizes, and replace the training data. The same pair can exhibit different degrees of sender dependence under different biases or training data; among the tested Qwen2.5 senders, stronger senders also produce fusion predictions that more often follow their answers.

1.1 Hold the model pair fixed and change receiver bias

For Qwen2.5-3B → Qwen3-1.7B, we subtract 20 from the receiver's logits for the B, C, and D answer tokens to favor A. Both language models remain unchanged. We retrain the fuser with this shift and apply the same shift during evaluation, including to the receiver-only baseline.

In the original configuration, sender zeroing reduces fusion accuracy below the receiver alone. With bias injection and retraining, zero-sender improvements appear:

Gain over the receiver with the same biasMMLU-ReduxOpenBookQA
First training run, zero sender+7.1 pp+2.4 pp
Second training run, zero sender+5.3 pp+4.2 pp

Changing receiver bias and retraining can produce sender-independent gains within the same model pair. We also cross training-time and evaluation-time bias conditions:

OpenBookQA accuracy (%).

Training conditionEvaluation biasReceiver onlyC2C: real senderC2C: zero sender
Trained without biasApplied55.827.627.6
Retrained with biasApplied55.873.858.2
Trained without biasNot applied65.676.261.2
Retrained with biasNot applied65.657.646.4

Table R3-1. Training and evaluation bias conditions affect fuser performance. Qwen2.5-3B → Qwen3-1.7B on OpenBookQA. Bias means subtracting 20 from the B, C, and D answer-token logits; the baseline has the same evaluation-time shift. This table shows one training run; the second run's zero-sender gains appear in the preceding table. pp denotes percentage points.

With bias present at evaluation, the fuser trained under that bias performs better; without it, that fuser performs worse. Its effect thus depends on the bias encountered during training.

1.2 Hold the receiver fixed and vary the sender

We fix Qwen3-0.6B and add Qwen2.5-1.5B and 3B between the existing 0.5B and 7B senders. Each fuser is retrained using the same multiple-choice training scheme.

SenderMMLU-Redux: real / zero senderOpenBookQA: real / zero sender
Qwen2.5-0.5B46.5 / 45.849.6 / 48.4
Qwen2.5-1.5B55.5 / 28.467.4 / 35.2
Qwen2.5-3B57.8 / 24.370.2 / 28.2
Qwen2.5-7B68.1 / 24.380.6 / 29.8
Receiver alone34.939.2

Accuracy (%). “Zero sender” denotes zeroing the sender cache before fusion.

Most of the 0.5B gain remains with the sender zeroed, while the other three pairings show stronger sender dependence. On questions where sender and receiver originally disagree, fusion follows the sender on 67%/75% of questions for 3B and 77%/77% for 7B, versus about 27%/26% for 0.5B (MMLU-Redux/OpenBookQA). Together with the cache ablations, this pattern is consistent with greater sender influence for the larger tested senders; prediction agreement alone does not establish causal transfer.

Changing the sender also changes its representations, so this within-series comparison does not completely separate ability from representation differences.

AuthorsOpenReview ↗

The same model pair exhibits different sender dependence under different training conditions (2/2)

Your concern (summary): Receiver bias, sender ability, and training conditions may affect whether C2C learns calibration or sender-dependent behavior, but the original experiments do not sufficiently disentangle them.

1.3 Hold the model pair fixed and compare training data

We compare Qwen2.5-7B → Qwen3-0.6B fusers trained on multiple-choice questions or general dialogue. Both use a 40,000-example cap and one epoch, but length filtering retains 39,867 multiple-choice examples and 38,603 dialogue examples, resulting in 1,234 and 1,195 training steps, respectively:

Training dataMMLU-Redux: real / zero senderOpenBookQA: real / zero sender
Multiple-choice questions68.1 / 24.380.6 / 29.8
General dialogue50.0 / 46.156.8 / 49.6

Accuracy (%). Receiver-only accuracy is 34.9% on MMLU-Redux and 39.2% on OpenBookQA.

With multiple-choice training, zeroing causes a large drop; with dialogue training, most of the gain remains. The same model pair therefore exhibits different degrees of sender dependence across these training conditions. Because filtering also changes training exposure, this comparison does not isolate data content from every other training difference.

These results connect the source of improvement to specific conditions rather than assigning every gain to either calibration or information transfer. We will specify what each intervention holds fixed and changes, and limit conclusions to the tested pairings and training settings.

AuthorsOpenReview ↗

Digit information remains readable from the fused cache despite incomplete answer recovery

Your concern (summary): Qwen2.5-7B → Qwen3-0.6B succeeds on color lookup and the first code digit but not the remaining digits. Is the information lost from the fused KV cache, or retained but inaccessible to the receiver?

Digit information remains linearly readable from the fused cache, but the receiver's predictions for the last three digits stay near chance under the paper's 1,000-step training conditions. Separately, the first digit is much easier to read near the end of the sender's prompt, offering a clue to the asymmetry.

Digit prediction accuracy (%); chance accuracy is 10%.

SenderRunsDigit 1Digit 2Digit 3Digit 4
Qwen2.5-0.5B39.39.810.49.6
Qwen3-1.7B398.899.299.099.0
Qwen2.5-3B482.410.310.09.8
Qwen2.5-7B198.99.810.210.7

Table R3-2. Teacher-forced per-digit accuracy after 1,000 steps. Receiver: Qwen3-0.6B; learning rate: 10⁻⁴. In the prompt, the sender sees the code and the receiver sees 0000. At each answer position we supply preceding gold digits and choose the highest-scoring token among 0-9. Run counts appear in the Runs column; the four 3B runs cover three seeds, with seed 0 run in two environments. Chance accuracy is 10%.

We probe the fused cache at the prompt positions of the four digits, training on 1,500 questions and testing on 500 separate questions. A separate linear classifier is fitted at each sampled layer; for each digit we report the highest observed test accuracy across those layers, then average over the four digits.

Measurement for Qwen2.5-7B → Qwen3-0.6BAccuracy
Teacher-forced digit prediction: first digitApproximately 99%
Teacher-forced digit prediction: each remaining digitApproximately 10%
Best-layer linear readout at digit positions in the fused cacheFour-digit mean: approximately 100%
Same probe setting, sender cache zeroedFour-digit mean: approximately 12%

Table R3-3. Teacher-forced digit prediction versus linear readout after 1,000 fuser-training steps. Probe results are means over all four digits, not only the last three.

These results rule out the simple interpretation that later-digit prediction fails because all corresponding information has disappeared from the fused representation. Moreover, the later digits remain near chance even with gold preceding digits, so accumulated prediction errors alone do not explain the failure. However, an entirely untrained fuser also gives nearly 100% mean readout accuracy. The fusion architecture itself therefore preserves some information, and successful probing does not establish that the trained fuser has learned effective use of it.

We separately probe the last sender-cache position passed to the fuser, near the end of the 7B sender's prompt. Unlike the preceding test, this attempts to read all four digits from the same position, using the same train/test split and best-layer reporting procedure:

DigitFirstSecondThirdFourth
Best-layer linear-probe accuracy99%32%19%17%

The first digit is much more accessible there, consistent with the prediction results. This suggests a possible route through the end-of-prompt representation, but does not distinguish a representation the receiver cannot readily use from failure to extract retained information at later answer positions.

Color lookup requires one answer letter and may benefit from the same route. This remains a hypothesis: we have not performed the corresponding linear-probe analysis.

We will report per-digit accuracy, exact-match accuracy, and linear recoverability separately, distinguishing information retained in the cache from information the receiver actually uses to answer.

Reviewer_GXYwOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper focuses on explaining the cache-to-cache (C2C) communication mechanism. This is an approach in which LLMs communicate by transferring their KV caches through a learned module known as a fuser. The fuser takes the KV caches of two models, say Model A and Model B, as input and produces a modified KV cache for Model B, which is then used to generate its answer.

The authors investigate four research questions concerning whether C2C transfers sender-specific information and what factors contribute to its performance improvements. For instance through experimentation, they suggest that the observed improvements in the released C2C configuration with the 0.6B receiver can largely be explained by correcting the receiver's output bias rather than transferring question-specific semantic information. They also investigate other model configurations to explore when sender-dependent information transfer occurs.

Critical strengths

Overall, I think the experiments and the phenomena investigated in this paper are interesting and worth studying.

I think the set of research questions (RQs) the authors aim to investigate is well-motivated and interesting, and the paper provides experimental evidence suggesting possible answers to these questions.

RQ2, which is investigated by replacing the sender's cache with zeros and showing that the accuracy improvement of the 0.6B receiver is almost entirely preserved compared with the real-cache condition, is indeed an interesting observation. I think this demonstrates that the correct sender's information is largely unnecessary for achieving the observed accuracy improvements, at least in this particular setting.

Furthermore, in RQ3, the authors decompose the fuser's effect on the output logits into a question-independent average component and a question-dependent residual. They show that, in the setting where C2C improves accuracy, the average component accounts for the majority of the improvement, which is again an interesting observation.

Critical weaknesses

1. Concerns with the generality and novelty of the proposed explanation

The strongest finding comes from the very small 0.6B model, which has an unusual preference for answer A. Among the released C2C checkpoints evaluated in Table 4, only the 0.6B receiver benefits from C2C. As such, the positive evidence for calibration as a source of C2C gains is concentrated in the 0.6B receiver, while other larger successful configurations in Table 5 behave differently. As such, this result does not establish bias correction as a general explanation across different receivers. It begs the question of how general this explanation really is. Without a proper theory or more experimentation, it is really hard to judge.

Further, the finding that C2C may not meaningfully use sender-specific content has already been investigated in prior work. For instance, Cheng et al. (2026) investigate a released C2C projector using correctly paired, deranged (mismatched-example), and zeroed sender caches. They find no detectable benefit from supplying the correctly paired cache compared with the deranged cache on their sender-private task, although zeroing the cache substantially reduces performance.

I think the main additional contribution here is to present output bias as an explanation. However, without further theoretical justification or more systematic experimental evidence, it is difficult to assess the generality and significance of this explanation especially with the results in table 4 and 5. This paper needs to show that this additional explanation is sufficiently general and significant to constitute a substantial new contribution compared to Cheng et al.

2. Need for better isolation of factors and stronger experimental robustness

I think the analysis in Table 5 does not really isolate the factors causing the observed effect. For instance, what really causes this effect? Is it the sender's strength, the receiver's bias, or some combination of both?

More isolated experiments would be:

  1. Fix Model B and vary Model A's strength, which is partially done when comparing 0.5B → 0.6B with 7B → 0.6B. However, to really investigate a phenomenon that is only being studied empirically, such two cases would not be enough.
  2. Fix Model A and Model B and vary B's answer-selection bias, and retrain the fuser in each case to better isolate the effect of bias.
  3. Use a more systematic set of sender–receiver combinations, multiple training seeds, and a variety of models.

Furthermore, the robustness of the newly trained fusers is insufficiently established, as most model-pair configurations are evaluated using a single training seed. This is particularly concerning where conclusions regarding representation mismatch or transfer mechanisms depend on unsuccessful training outcomes, for example in Table 16.

Only two of the five newly trained configurations in Table 5 are repeated with a second seed. This is rather limited for establishing a general mechanism. More repeated training runs and statistical analysis of the variability across training seeds would strengthen the experimental conclusions.

3. Clarification regarding the mechanistic explanations and mathematical claims

Representation mismatch and information transfer (RQ1). The paper claims that the reason for failure from 0.5B → 0.6B in Table 1 is representation mismatch, but it is not really obvious how this follows from the experiment. Why is it not a failure of the training procedure or something else?

There should be a more controlled experimental design to investigate whether this is indeed the reason, for instance, testing different optimization baselines, alternative learning rates, and a more systematic range of training budgets. I recognize that the authors already investigate additional training steps and gate settings in Appendix G, but this does not fully isolate the cause of failure.

Also, in Table 16, the authors acknowledge that the experiment itself demonstrates some cross-family information transfer, particularly in the case of the first code digit. However, they do not fully explain why the remaining digits fail to transfer. Could this be due to limitations of the training procedure or multi-token generation rather than representation mismatch?

Furthermore, zero exact-match accuracy does not necessarily mean no information transfer. The results in Table 16 demonstrate that information about the first code digit can cross between different model families, even when the receiver fails to reconstruct the complete four-digit code. Failure to reconstruct all four digits does not, by itself, identify representation mismatch as the cause. I think the authors should investigate this partial-transfer behavior more thoroughly before attributing the failure primarily to representation mismatch.

Mathematical claim regarding constant shifts. The paper claims: " A constant shift can raise accuracy only if the receiver has a constant bias."

I would appreciate clarification of this statement. For instance, consider the following example.

We have four questions with correct answers (A, A, B, B), and the receiver predictions are (B, A, A, B), with the difference between the scores of A and B for each question being (-0.1, 2, 1.9, -3.8).

So there is no marginal preference for A or B, and the accuracy is 50%.

Now add 0.2 to the score of A for every question. The accuracy increases to 75%.

Therefore, a constant logit correction can improve accuracy even when the receiver's marginal prediction distribution is perfectly balanced. If the authors mean that a skewed answer-selection distribution is necessary for such improvements, this example provides a counterexample. It would be great if further clarification of what is meant by "constant bias" is provided and why this condition is claimed to be necessary.

Finally writing and readability. The paper is rather hard to read. It feels more like many different interesting experiments being pushed next to each other, rather than a coherent investigation of one central problem. The existence of many different factors, model configurations, and experimental settings makes it difficult to follow the paper as a whole.

There are also specific readability issues. For instance, the caption of Table 16 is very long and contains confusing descriptions such as "init. from §4.4 fuser". The receiver model is not even explicitly mentioned in the caption, requiring the reader to look through previous sections to understand the experimental setup.

I think the paper would benefit a lot from focusing on perhaps the two most interesting observations or experiments and building a more coherent investigation around them, rather than presenting many different experiments without sufficiently developing their connections.

Overall Concern

The paper does not sufficiently establish that the identified calibration mechanism generalizes across independently trained fusers, model pairs, or training conditions.

Consequently, the experimental evidence is substantially narrower than the broader mechanistic perspective suggested by the paper. Without further theoretical justification or more systematic controlled experiments, the paper seems more like a localized empirical study than a general investigation of C2C communication.

Questions for authors

I think addressing the concerns and weaknesses pointed out above would be a good place to start. I would also like to suggest some additional experiments.

  1. Generality: What evidence supports the generality of the proposed calibration explanation beyond the released 0.6B receiver, particularly given that the larger released receivers do not benefit from C2C? Are we trying to give explanation for a specific case or a more general phenomenon.

  2. Representation mismatch: What evidence establishes that the failure of the 0.5B → 0.6B configuration in Table 1 is caused by representation mismatch rather than optimization or training limitations? Could the authors clarify why partial cross-family transfer succeeds in Table 16 while complete code reconstruction fails?

  3. Isolation of factors: How can the authors distinguish the effects of receiver bias and sender strength on whether the fuser learns calibration or sender-dependent transfer?

Additionally, I would like to see more systematic experiments across independent training seeds and learning rates. Again I recognize that Appendix G already investigates additional training steps, but this does not fully characterize training variability.

10 comments
AuthorsOpenReview ↗

Prompt-induced answer bias helps explain differing results across the released configurations

Your concern (summary): The positive evidence for calibration is concentrated in the small 0.6B receiver. Its generality, including the different outcomes for larger released configurations in Table 4, remains unclear.

Thank you for raising this concern. We added interventions on all three released configurations and a bias survey of 21 open models. Calibration explains a substantial part of the gain in the released 0.6B configuration, but its applicability cannot be determined from model size alone. Larger models can also be affected on difficult tasks, while smaller models need not have equally large calibration headroom.

1.1 Three released configurations

We use the released fusers for Qwen2.5-0.5B → Qwen3-0.6B, Qwen2.5-1.5B → Qwen3-1.7B, and Qwen2.5-7B → Qwen3-8B, without further training.

The formatting instruction “The correct answer is A/B/C/D” biases some receivers toward the first listed letter. For example, an OpenBookQA question asks what plants require to grow, with choices A. acid rain, B. pesticides, C. shafts of sunlight, and D. moonbeam rays. The correct answer is C. Qwen3-0.6B alone selects A under the original instruction, D when only the instruction's letter order is changed to D/C/B/A, and B with B/C/D/A. Removing the formatting sentence makes it select C. This is not an isolated example: on the 500-question set, it selects A on 79% of questions under the original instruction and D on all questions under D/C/B/A.

We remove only this sentence and compare C2C with the receiver alone under each prompt. All C2C results in the following table use a zeroed sender cache, so the effects do not require question-specific sender information.

ReceiverC2C gain with format instructionC2C gain without format instruction
Qwen3-0.6B+11.8 pp+2.0 pp
Qwen3-1.7B-5.6 pp+0.4 pp
Qwen3-8B+0.4 pp-0.2 pp

OpenBookQA, using the paper's option-score evaluation. Each gain is relative to the receiver alone under the same prompt. pp denotes percentage points.

The effects differ in direction. For 0.6B, fusion reduces A selections from 79% to 38% and improves accuracy. For 1.7B, it increases A selections from 32% to 49% and reduces accuracy. Removing the formatting instruction substantially reduces both effects. The 8B receiver is less affected on this benchmark and changes little after fusion. Thus, prompt-induced answer bias helps explain why Table 4 contains improvements, degradations, and near-zero changes.

With the original formatting instruction retained and the sender cache zeroed, we also calibrate the receiver-only and C2C logits separately. For each letter, we subtract its mean logit on the other half of the evaluation questions, swap the halves, and average accuracy over 20 random splits. This uses unlabeled evaluation-set statistics. After calibration, the 0.6B C2C gain falls from 11.8 to 4.5 pp; the 1.7B and 8B differences become -1.6 and -1.5 pp. Calibration substantially reduces the 0.6B gain and the 1.7B degradation, while the 8B differences remain small. The remaining 4.5 pp for 0.6B also occurs with a zeroed sender cache and therefore does not require question-specific sender information.

AuthorsOpenReview ↗

Prompt sensitivity varies across models and tasks and does not determine calibration headroom

Your concern (summary): The positive evidence for calibration is concentrated in the small 0.6B receiver. Its generality, including the different outcomes for larger released configurations in Table 4, remains unclear.

1.2 Bias depends on the model-task combination

We evaluated 21 open models independently, without C2C. Five completed the seven task groups shown below. For each question, we compare A/B/C/D and D/C/B/A in the formatting instruction while keeping the question and actual choices fixed. Among questions whose predictions change, a change counts as following the instruction if the original prediction is A or the reversed-instruction prediction is D. A change counts in the opposite direction if the original prediction is D or the reversed-instruction prediction is A. We subtract the latter count from the former and divide by all questions to obtain the net following rate.

Net following rate (% of all questions).

ModelARC-COpenBookQAMMLU-ReduxMMLU-ProSuperGPQA EasySuperGPQA MediumSuperGPQA Hard
Qwen3-0.6B86898797979899
Qwen3-1.7B18272740364666
Qwen3-4B43610101015
Qwen2.5-7B0-214449
Qwen3-8B-1-1-12977

Table R2-1. Sensitivity to the first letter in the formatting instruction. Receiver-only evaluation; values are net following rates (% of all questions), rounded to integers. Negative values retain their sign. The instruction lists A-D even for tasks with more than four answer options; models still choose among all actual options. This measures prompt sensitivity, not calibration gains.

Two empirical patterns emerge:

  • A given model often shows greater sensitivity on harder task groups. Qwen3-1.7B rises from 18% on ARC-C to 66% on the hard SuperGPQA group; Qwen3-4B rises from 4% to 15%.
  • Models differ substantially on the same task. On OpenBookQA, the rate is 89% for Qwen3-0.6B, 3% for Qwen3-4B, and near zero for the 7B and 8B models. On SuperGPQA, the latter models also show rates of approximately 4%-9%.

Task groups are ordered by the evaluated models' mean accuracy, not by an independent, precise difficulty scale. Differences in the number of answer options are another confound. We therefore present these as empirical trends across models and tasks, not as a universal law that harder questions necessarily induce more bias.

1.3 Sensitivity is not the same as calibration headroom

The table cannot be read as the amount of recoverable accuracy. On OpenBookQA, Gemma-3-1B has a net following rate of 98%, yet a fixed score shift improves accuracy by only about 1.4 pp. Qwen2.5-1.5B shows almost no first-letter following but gains about 5.1 pp from a fixed shift. Here, shifts are fitted using answer labels on one subset and evaluated on held-out questions; this differs from the label-free mean-subtraction control in part 1.

For model-task combinations with measurable calibration headroom, calibration is a candidate explanation that should be tested, not an automatically established C2C mechanism. Establishing its role also requires sender ablations and output-score analysis. We will not generalize the released 0.6B explanation to all configurations.

AuthorsOpenReview ↗

Interventions explain benchmark gains and identify the scope of the calibration explanation

Your concern (summary): Sender-content insensitivity overlaps with Cheng et al. The additional output-bias explanation needs systematic evidence of its importance and scope, particularly in light of Tables 4 and 5.

Our contribution is to explain the source of reported benchmark gains and the conditions under which that explanation applies. Reproducing sender-content insensitivity in a released configuration would not, by itself, establish this contribution.

Cheng et al. tested the released Qwen2.5-7B → Qwen3-8B configuration on sender-private constructed tasks and found no detectable advantage of correctly paired over mismatched caches. This overlaps with our RQ1, but does not analyze C2C's reported public-benchmark gains. Using our paper's option-score evaluation, we find that the released Qwen2.5-0.5B → Qwen3-0.6B improvement persists without sender content; output-bias analysis addresses this remaining improvement.

We also adopted their mismatched-cache control to reduce the concern that zero inputs depart from the normal cache distribution. Across four benchmarks for the released 0.6B configuration, correctly paired and mismatched caches differ in accuracy by less than 0.5 pp, consistent with the zero-cache observation.

For Table 4, the prompt interventions in our response on generality connect accuracy changes to a specific answer bias. Removing the formatting instruction reduces the zero-sender effects from +11.8/-5.6/+0.4 pp to +2.0/+0.4/-0.2 pp for the three receivers. This goes beyond observing that a gain survives sender removal.

For Table 5, additional training experiments identify boundaries of the explanation. With Qwen2.5-3B → Qwen3-1.7B fixed, injecting an A-favoring bias and retraining produces zero-sender MMLU-Redux gains of 7.1 and 5.3 pp across two seeds. For Qwen2.5-7B → Qwen3-0.6B, the real/zero-sender MMLU-Redux accuracies are 68.1/24.3 after multiple-choice training and 50.0/46.1 after dialogue training, against a 34.9% receiver baseline. Both use a 40,000-example cap and one epoch, but filtering retains 39,867 versus 38,603 examples; this is not an exact match in training exposure. The bias and sender-strength comparisons are detailed in our response on factor isolation and training robustness.

Our additional contribution is an intervention-based explanation of reported benchmark gains and its scope. Calibration explains a substantial portion of the released 0.6B gain, while other configurations depend on receiver bias and training conditions. We will narrow our claims accordingly.

Reviewer_m8W4OpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

this paper reproduces C2C and investigates the source its accuracy gain. It does this by remove sender cache, run synthetic information transfer tasks, and logit-shift decomposition. The results suggests that C2C gains is most explained by correcting receiver's pre-existing bias. Most significantly, replacing sender's cache with zeros preserves nearly all improvements, and a single question-independent logit correction reproduces or exceeds the original gain.

Critical strengths
  1. The logit shift result is significant mechanistic probing result.

  2. The result shows significant practical implications for multi agent LLM systems.

Critical weaknesses
  1. The claimed novelty overlaps partially with existing causal audits of latent communication, such as "When Does Latent Communication Pay? A Causal Audit of Relayed KV Caches in Multi-Agent LLMs". The author may want to revise the claim to be narrower.

  2. The experiments are limited and not yet show generality across architecture, evaluation settings, and training distributions.

Questions for authors

N/A

5 comments
AuthorsOpenReview ↗

Our contribution explains benchmark gains and the limits of the calibration explanation

Your concern: The claimed novelty partly overlaps with prior causal audits of latent communication, including When Does Latent Communication Pay? A Causal Audit of Relayed KV Caches in Multi-Agent LLMs. The contribution should be stated more narrowly.

Thank you for pointing this out. We will narrow our contribution claims and clarify the distinction from Cheng et al. The two studies overlap in the diagnostic approach used in RQ1: making the answer available only to the sender and testing whether the receiver can recover it through the incoming KV cache.

Our additional question is where C2C's reported improvements on public benchmarks come from. Cheng et al. evaluated the released configuration with an 8B receiver on constructed tasks, but did not analyze the benchmark gains reported for C2C. We evaluate the released weights on these benchmarks using the option-score protocol described in our paper, then test whether the observed gains require sender content and analyze the improvements that remain without it. We also check free generation in part 3 of our response on generality.

Our main contribution is therefore an explanation of reported benchmark gains: for the released Qwen2.5-0.5B → Qwen3-0.6B configuration, sender ablations and output-score decomposition show that correcting the receiver's output bias can explain a substantial portion of the improvement. We also train additional model pairings and vary training data to identify conditions with stronger sender dependence, thereby delimiting the scope of this explanation.

AuthorsOpenReview ↗

Calibration explains gains in some configurations, but not all

Your concern: The experiments are limited and do not yet establish generality across architectures, evaluation settings, and training distributions.

Thank you for this suggestion. We added comparisons across model pairings, training data, prompts, and answer-reading procedures. The results show that the calibration explanation does not extend uniformly to every pairing or training condition. The controlled pairing comparisons below remain within the Qwen series.

1. Model architectures and pairings

We added Qwen3-0.6B → Qwen2.5-0.5B, extending the receiver from Qwen3 to another generation of Qwen models, with two training seeds. C2C did not improve receiver accuracy in either run. With the sender cache zeroed, one run showed an improvement and the other did not. We therefore do not treat this result as evidence that the calibration mechanism is established across architectures.

With the Qwen3-0.6B receiver fixed, we also added 1.5B and 3B senders between the existing 0.5B and 7B senders to examine sender strength more systematically. We will explicitly state that these comparisons are limited to Qwen pairings.

2. Training distributions

We compare separately trained fusers for the same Qwen2.5-7B → Qwen3-0.6B pair, using multiple-choice questions or general dialogue. Both use a 40,000-example cap and one training epoch. Length filtering retains 39,867 multiple-choice examples and 38,603 dialogue examples, so the effective training exposure is not exactly matched. These are distinct from the released weights used in the prompt experiments in parts 2 and 3 of this response.

Training dataMMLU-Redux: real / zero senderOpenBookQA: real / zero sender
Multiple-choice questions68.1 / 24.380.6 / 29.8
General dialogue50.0 / 46.156.8 / 49.6
Receiver alone34.939.2

Accuracy (%). “Zero sender” means that the sender cache is replaced with zeros before fusion.

Zeroing the sender causes a large accuracy drop after multiple-choice training, whereas most of the improvement remains after dialogue training. Thus, the same model pair can exhibit different degrees of sender dependence under different training distributions; calibration is not a uniform explanation for every training condition.

AuthorsOpenReview ↗

The formatting instruction lowers receiver-only accuracy and inflates C2C’s apparent gain

Your concern: The experiments are limited and do not yet establish generality across architectures, evaluation settings, and training distributions.

3. Evaluation settings

Under the option-score evaluation used in our paper, removing a formatting instruction substantially improves the receiver-only baseline and sharply reduces C2C's relative gain. We also examine free-generation evaluation in part 3 of this response.

In addition to the question and choices, the original prompt includes:

Respond ONLY in the following format: “The correct answer is A/B/C/D”.

Although intended only to specify the response format, this sentence affects answer selection. We keep the questions, options, and correct answers unchanged and modify only the letter order in this instruction. On 500 OpenBookQA questions, the receiver selects A on 79% of questions with A/B/C/D, but selects D on every question with D/C/B/A. When B or C is listed first, it selects that letter on approximately 96% or 94% of questions, respectively. The predictions strongly follow the first letter in the formatting instruction. Here, selection follows the paper's evaluation procedure: append “The correct answer is” and choose the highest-scoring answer letter.

We then remove the entire formatting sentence and compare receiver-only and C2C accuracy under identical prompts and answer-reading procedures. C2C's gain is their accuracy difference.

Accuracy (%).

BenchmarkFormatting instructionReceiver onlyC2C: real senderC2C: zero sender
OpenBookQAKept39.251.651.0
OpenBookQARemoved47.250.049.2
ARC-CKept39.153.955.0
ARC-CRemoved52.153.052.8
MMLU-ReduxKept34.942.443.2
MMLU-ReduxRemoved44.742.643.6

Table R1-1. Removing the formatting instruction improves the receiver baseline and reduces C2C's relative gain. Released Qwen2.5-0.5B → Qwen3-0.6B weights, without additional training. All results use option-score evaluation: 500 OpenBookQA questions, the 1,165 four-option ARC-C questions, and 5,632 MMLU-Redux questions. Each benchmark compares the receiver alone, C2C with the real sender cache, and C2C with the sender cache zeroed.

C2C accuracy changes little; the main change is that the receiver performs better on its own. On OpenBookQA, receiver-only accuracy rises from 39.2% to 47.2%, reducing the gap to real-sender C2C from 12.4 to 2.8 percentage points. A substantial part of the apparent gain under the original prompt therefore corresponds to accuracy lost by the receiver because of the formatting instruction.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.