acceptodds
Withdrawn from ICLR 2027

Augment the Cache, Replace the Activation: A 2x2 Study of Internal-State Sharing

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Multi-agent systems built from frozen language models communicate through text, which discards the internal state the sender already computed and makes the receiver pay to rebuild it. Work on internal channels avoids that cost, framing the problem as transfer: agent A sends information to agent B. Every agent pair that must communicate needs its own transfer bridge, and every agent added to a system needs a new bridge fitted against each previous agent. We reframe this problem as sharing: an agent publishes its internal state once into a pool, and every peer reads that pool through a single mechanism fitted to no sender in particular, so adding an agent adds a publisher and no new bridge. Whereas the carrier of the internal state and the fusion mechanism are usually chosen independently in the literature, our main finding is that these two choices are coupled; the deciding factor is where the carrier enters the receiver’s computation. Consumability is a property of the receiving computation rather than of the representation. We find that the key–value (KV) cache must be augmented rather than swapped in: the receiver cross-attends over a peer’s cache while keeping its own, and with only a small query-side adapter trained, matches the accuracy it gets from reading the source document. Activations invert this. They are best written straight into the receiver, with no bridge, no adapter and no training, which beats exposing them alongside the receiver’s own, though that advantage rests on knowing where in the receiver the state belongs. The rule is augment the KV cache, replace the activation, established under one protocol with a shared control ladder. One adapter, trained once, also serves publishers it never trained on, and a pool of several at once, but only those whose internal state lies close to the receiver’s own.

Primary area:
foundation or frontier models, including LLMs
Keywords:
multi-agent systems, inter-agent communication, residual stream activations, activation injection, KV cache sharing, cross-attention, parameter-efficient adaptation, frozen language models, latent communication, information sharing

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_hzWaOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies how LLM agents in multi-agent systems could communicate through their internal states instead of text. They compare sharing the KV cache with sharing the residual-stream activations, by either replacing the receiver's state or augmenting it. They claim that the KV cache works only if augmented, while the activations work if they replace the receiver's. They explain this with where each of the two enters the receiver's computation, i.e., through the softmax for the keys and through the residual for the activations. To validate their hypothesis, they use a single task, where two copies of Qwen2.5-1.5B read a SQuAD passage, one of them with the answer masked, and test whether the internal state of the copy that read the full passage lets the other one answer.

Critical strengths

Communication between agents through internal states rather than text is an interesting direction, and understanding which internal state should be shared and how the receiver should use it is a relevant question. The paper makes a hypothesis to characterize the effectiveness of two methods (augmentation and replacement) on two different means of transfer (KV cache and residual-stream activations). If the empirical evidence solidly confirmed this hypothesis, that would be a meaningful contribution to the latent communication literature.

Critical weaknesses

First of all, I found the paper very hard to follow, because it introduces a lot of jargon (e.g., "readout map", "tolerable set", "consumption point") and the main table mixes different metrics and sample sizes across columns.

The empirical evidence does not clearly support the main claim of the paper, i.e., "augment the KV cache, replace the activation".

  • Holder and receiver are the exact same model, therefore the holder's state is exactly the state the receiver would have computed on the full passage. So overwriting the activations at the answer positions, simply gives back to the receiver the states it would have had if it had read the answer, i.e., it is activation patching, which on the same model is expected to restore the output. Similarly, KV augmentation lets the receiver attend over the cache it would have computed itself on the full passage. Also, the span-copy rate the paper uses to rule out copying only counts answers that are correct and have at least two tokens, so it can never be higher than the accuracy itself, and the gap between the two does not tell whether the receiver is copying.
  • The 2x2 is not a fair comparison. Looking at the released code, the KV substitution replaces the keys and the values of every token, including the question, at every layer, with those of the nearest tokens from other passages, while the activation replacement only overwrites the answer positions with the exact activations of the holder. So it is not surprising that the KV substitution does even worse than giving the receiver no help at all. This is far from the "conditions engineered to be maximally favorable to substitution" claimed by the authors: the substituted keys and values come from different tokens, contexts and positions, so nothing guarantees they can work together. Also, the two "augmentation" cells are different operations, so they do not test the same mechanism.
  • The paper ranks the carriers as "activations over values over keys", but this comparison is not fair as well. Replacing the activation is the same as replacing both keys and values at all the following layers, so it should be compared with keys and values replaced together, which is only reported at a single layer, where it is not comparable. Replacing the keys alone is expected to fail, since they only decide where to attend, while the content is in the values.

This is important because the main contribution of the paper is not convincingly supported by the proposed experimental setup.

Anyways, the positive results are restricted to a single model and are weak.

  • The paper shows that sharing between different model families performs badly on both the KV cache and the activations, and the positive results only refer to a single model, so the motivation of the paper (many agents sharing a pool) is not supported, as the authors admit in the Limitations section.
  • Everything is measured on a single artificial task, i.e., recovering a masked answer in a SQuAD passage, with a single seed, and the activation replacement relies on the answer positions, which are taken by comparing the masked and the full passage, and which a real agent would not know.
  • The paper is motivated by the idea that agents should communicate in latent space, so that the receiver does not need to rebuild the context by reading text, but the cost of reading the text is never compared with the cost of the proposed methods.

This is important because, even assuming the experimental setup supported the paper's claims, the empirical evidence is not solid enough to show that the proposed hypothesis holds beyond a single model and task.

Questions for authors

My main concerns require new experiments and a substantial rewrite, so I do not expect a rebuttal to be enough to change my recommendation. Still, the following points could affect my assessment.

  • Could you compare the two carriers under the same conditions? I.e., a KV substitution restricted to the answer positions, with keys and values replaced together from the same layer on, and both carriers perturbed in the same way and at the same fidelity. If the KV cache still fails while the activations work, this would support the main claim.
  • You show that sharing between different model families performs badly on both KV cache and activations, but you only fitted a projection on 40 items, while there is evidence that different models can communicate through learned projections between their representation spaces. As you also say, the bottleneck is the alignment, so this negative result does not tell whether the hypothesis holds across models. Could you test it with a stronger alignment, e.g., a trained projection?
  • I am aware this is not something that can be fully addressed in a rebuttal, but to be convinced I would need at least one more task, e.g., one where the information to share is not a short span at known positions. Could you show that the results hold on such a task, even at a small scale?
Reviewer_CxYsOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether internal states can be shared between frozen language models, comparing two carriers (KV caches and residual activations) crossed with two mechanisms (replacement and augmentation) in a 2x2 design. Using a split-context SQuAD probe, the paper reports that KV-cache augmentation and activation replacement transfer answer information, while key substitution fails. Additional experiments examine source controls, shared pools, held-out senders, and architectural variation.

Critical strengths
  1. The controlled carrier-versus-mechanism comparison addresses a useful question, and the experimental design includes valuable safeguards: native-forward parity, no-op, mis-source, and answer-disjoint controls.
  2. Negative results and limitations are reported candidly. The seven-receiver substitution sweep and the held-out-sender tests help delimit the compatibility regime rather than relying on a single successful same-model case.
Critical weaknesses
  1. The central measurements need reconciliation. Table 1 reports 0.180 at alpha=0, although Section 4.1 describes the interpolation as starting from the true key and reports 0.665 for the identity case. Tables 3-4 label activation replacement as reaching the native ceiling by comparing a conditional accuracy of 0.742 against an unconditional native reference of 0.742, while conditioning on items the native reference answers correctly would make the reference's conditional accuracy 1.0, not 0.742. At layer 12, shared-fact accuracy is only 0.567. As written, the headline comparison is ambiguous.
  2. Proposition 1 appears overstated. For a fixed transformer probability readout, vanishing cache MSE implies convergence of the output distribution under standard continuous divergences, since softmax depends on logit magnitudes rather than only their ranking. Observed greedy-answer failures do not establish that arbitrarily small MSE can coexist with maximally different probability readouts. The claimed mechanism needs a precise statement and a supporting derivation.
  3. The broad preference for replacement over augmentation is not cleanly isolated. Activation replacement uses oracle positions and layers while augmentation mixes three holders, and the positive results largely come from one model and one factual probe. A comparison matched on source, injection site, and metric is needed to support the general rule, even within the acknowledged compatibility regime.

These measurement ambiguities and the unsupported mechanistic claim currently affect the central conclusion, and resolving them could make the controlled empirical findings a useful contribution.

Questions for authors
  1. What exactly is interpolated in Table 1, and why does its zero-displacement endpoint differ from the identity case? Please also align the conditional and unconditional metrics and references in Tables 3-4.
  2. Can Proposition 1 be qualified to specify the fidelity measure, divergence, and limiting regime, with a proof that distinguishes changes in probability from discontinuities in greedy generation?
  3. Does replacement still outperform augmentation when both use the same holder, injection sites, evaluation subset, and metric?
Reviewer_TYmnOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper investigates latent communication among frozen LMs, extending pairwise state transfer to a sender-agnostic "sharing" setup where a receiver reads from a shared state pool. The central finding is that the choice of state carrier (KV cache vs. residual activations) and fusion rule (substitution vs. augmentation) are tightly coupled by the receiver's computation graph: because keys pass through a competitive softmax, they tolerate far less perturbation than residual activations, which propagate along a near-unit-gain linear path. Using a split-context fact-transfer probe on SQuAD (primarily with a frozen Qwen2.5-1.5B receiver), the authors show that key substitution fails across seven models from three families, even when using nearest-neighbor native keys. By contrast, KV augmentation works zero-shot, matches the native baseline with a q/k/v LoRA bridge, and scales to multi-holder pools and unseen senders. Conversely, residual activations favor direct oracle replacement over augmentation, distilling into the core rule: "augment the KV cache, replace the activation." Finally, cross-checkpoint tests show that the learned bridge transfers to instruction-tuned variants, but fails on domain-specialized (code/math) checkpoints and across model families.

Critical strengths
  1. This paper separates substitution from augmentation, together with the receiver-centric readout map, and gives a common vocabulary for KV-level and activation-level methods that prior work tends to mix.
  2. The paper states its main limitations directly: oracle position and layer for activation replacement, a single seed, a non-significant margin over the native reference, and the failure of the mismatch control at L≥16L \geq 16.
Critical weaknesses
  1. Section 1 is hard to follow. The introduction and related work are merged, so the paper’s own question appears only after a long list of citations. The main motivation gets two sentences and defers all of its evidence to the appendix. The contribution list uses terms such as "consumability", "readout gain", and "conditioning cliff" before Section 2 defines them.
  2. All positive findings depend on a single receiver (Qwen2.5-1.5B), whereas the architecture sweep and cross-family tests rely on legacy models like GPT-2, Pythia, and GPT-Neo-1.3B. By comparison, closely related works such as LatentMAS [1] and Cache-to-Cache [2] already build on Qwen3 backbones (e.g., Qwen3-0.6B). Furthermore, testing cross-family sharing solely with GPT-Neo-1.3B makes it difficult to disentangle true architectural incompatibility from legacy artifacts like outdated tokenizers and pretraining data. The authors should expand their evaluation to contemporary small models, such as Qwen3/3.5, Gemma 3/4, and Llama 3.2.
  3. Evaluating solely on SQuAD with short verbatim spans (≤4\le 4 words) reduces the problem to passive context reuse (akin to distributed RAG) rather than true multi-agent communication, as the holder performs no reasoning. Prior latent-communication baselines typically benchmark on reasoning-heavy tasks (e.g., GSM8K, MedQA, HumanEval-Plus) or multi-player coordination games. The current results do not show whether the proposed carrier-fusion rule generalizes to settings where the transmitted state encodes abstract or multi-step reasoning.
  4. One receiver-side bridge transfers only to the instruction-tuned checkpoint (cosine 0.9994). It fails on the coder and math checkpoints and on GPT-Neo. The argument that sharing needs one bridge per receiver instead of one per pair therefore holds only for near-identical checkpoints. A similar argument has already been made for heterogeneous models by prior work, Vision Wormhole [4].

[1] Latent Collaboration in Multi-Agent Systems, ICML 2026 [2] Cache-to-Cache: Direct Semantic Communication Between Large Language Models, ICLR 2026 [3] Communicating Activations Between Language Model Agents, ICML 2025 [4] Vision Wormhole: Latent-Space Communication in Heterogeneous Multi-Agent Systems, arXiv 2026.

Questions for authors

Check the weaknesses above.

Comments on the paper

1 comment

We thank the reviewers and the area chair for their time and constructive feedback. After careful consideration, we have decided to withdraw the submission and use these comments to substantially improve the work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.