- Rating
- 2(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This work presents receiver-conditioned latent communication to mitigate the heavy memory and context overhead of full KV-cache transfer in multi-agent LLM systems. It proposes CacheBack, a training-free attention-driven selection mechanism that uses the receiver’s query to extract and transmit task-relevant sender states. Integrating query-aware position selection with compact state representations, CacheBack cuts communication cost while retaining critical information.
Critical strengths
This work features a formulation that addresses the scalability bottleneck of context re-expansion in full-KV-cache communication via receiver-conditioned selection without extra training. CacheBack adopts a lightweight, easy-to-implement design by correcting mean query attention to highlight highly attended positions, extending existing KV-cache selection strategies.
Critical weaknesses
-
Methodological novelty is very constrained: CacheBack adapts existing KV-cache compression to multi-agent communication via receiver-query conditioning and heuristic attention reweighting. The paper needs clearer differentiation from prior query-aware KV-cache selection and stronger justification for the scoring function.
-
Experimental scale and statistical validity are limited. Evaluations adopt merely 50 samples per benchmark, and only the Easy partition of LongBench v2 is tested due to text truncation issues, weakening generalizability. More difficult tasks, larger test sets and competitive baselines are required to solidify conclusions.
-
Practical deployment constraints lack sufficient investigation. The method requires aligned sender-receiver representations and is validated mainly on high-bandwidth multi-GPU environments. Its performance under heterogeneous models, variable network conditions and dynamic receiver queries remains not clear.
Questions for authors
See Weakness.
2 comments
Reviewer 3 (2/2)
Weakness 3: Practical deployment constraints
Network costs
Network transfer costs in KV-based communication are one of the motivations for incorporating CacheBack in these multi-agent systems. CacheBack simultaneously reduces bytes over the wire compared to transmitting full KV caches (§1, §2) by reducing the number of positions sent and by sending one continuous input row per position instead of its KV position. This is discussed heavily in §3.1.2, where we show that for a Qwen 3 8B, transmitting a continuous input row is 18x smaller per position than a KV position. The receiver then reconstructs the activations through a fresh prefill.
Across 3.1.2 and Appendix C.3, we address communications over lower bandwidth networks. Across our Qwen experiments, the average size of a message is 25MB, which is negligible to decoding latency on high bandwidth networks (NVLink)with peak transfer times of 55.4us. Figure 8, shows that this would fall to 0.755us if replaced with tokenIDs where possible. We also see that with lower network speeds 300Mbits/s peak transfer time would fall from 664ms to 9.06ms under token IDs. Both transfer times are small when compared with the time spent generating text reports.
In §5 we also briefly discuss the tradeoff of CacheBack latency and how the advantage narrows when network transfer or receiver prefill costs more than the sender decoding avoided.
Heterogenous models
CacheBack uses LatentMAS for latent reasoning, which we keep fixed as stated in §2. LatentMAS was designed for the homogeneous setting stated in §1. This enables representation compatibility without a learned cross-model adapter. As §5 states, this allows us to isolate whether receiver-conditioned message selection can prevent context re-expansion without degrading communication quality and while reducing total serving costs.
However, we ran another ablation where we did not pass latent reasoning but used CacheBack to compress context (via indexing) between Nemotron and Qwen.
For 8x compression on FanoutQA (n=50)
| Sender → Receiver | Text | CacheBack | Δ vs text [95% CI] |
|---|---|---|---|
| Qwen3-8B → Nemotron-12B | 0.38 (88 s) | 0.49 (29 s) | +0.11 [0.00, +0.22] |
| Nemotron-12B → Qwen3-8B | 0.42 (65 s) | 0.55 (49 s) | +0.13 [+0.02, +0.25] |
Latency reflects prefill and decoding only, using the paper’s precomputed results for a fair per-message comparison. It does not include the total system time reported in the paper.
Dynamic Receiver queries.
We draw the reviewer’s attention to §5, where we explain why CacheBack can support different receiver queries at inference time. To test this on the 50 FanOutQA questions, we prefilled all of the required documents once for Qwen3 8B. Then ran 11 queries (the original plus 10 generated from the model) at CacheBack 8x compression.
The CacheBack selection algorithm is only 0.7s, therefore after we first prefill a sender’s context we can repeatedly re query the same cache cheaply. In our setup prefilling all 150k tokens takes 7.4s for the combined three senders. The optional latent steps add 5.4s and currently run outside the serving engine at batch size on. Forking vLLM and allowing cache mutation would drastically reduce this.
In this setup, Qwen 3 8B generated 10 independent queries for each question. The end-to-end accuracy showed no significant difference between the original (52.7) and the average of the 10 independent queries (51.5). The range across the 10 queries was 46.7 to 56.7.
Reviewer 3 (1/2)
We thank the reviewer for a thoughtful review, and we appreciate the time and effort it takes to review papers. Below, we address the main concerns raised and will make the changes in the final version of the paper. Our main comment is that Cacheback is a simple, drop-in, training-free method that 1) addresses context-limit challenges in communication, and 2) outperforms existing methods across a variety of open attention-based model architectures.
Weakness 1: Methodological novelty and the case for attention correction
Receiver Conditioned Latent Communication (RCLC) addresses the context-window bottleneck created by combining the entire worker's state into the receiving agent, contradicting the context management benefit of delegation. Our key insight is that we can separate what we send versus how we represent it. What we send is therefore centered around the receiver’s information needs and context window (§2, Eqs. 1–2; Appendix A).
We then propose CacheBack, a training-free implementation of this formulation that builds on SnapKV’s attention-based position selection. We then hypothesize a specific failure of mean query attention where useful sender tokens that receive strong attention from an individual query token are evicted due to low values from other query tokens (Figure 3). CacheBack is designed to address this failure by upweighting concentrated attention scores. We test whether this correction improves recall on a heldout set of questions excluded from the end-to-end evaluation (§3.2; Appendix D).
To show the significance of this correction term, we ran an additional end-to-end ablation comparing mean query attention to cacheback on FanoutQA n=50 for Qwen 8B. Same-size text workers score 40.7%.
At tested ratios from 16x compression onwards, we see CacheBack outperforms mean query attention, highlighting the significant impact of our correction method on downstream task accuracy. This now supplements the initial selector comparison shown in §3.2 and Appendix D.1, where we compare CacheBack with mean query attention, StreamingLLM, H2O, ChunkKV, and KVzip.
Weakness 2: Experimental scale and statistical validity
We agree that ideally the tests would have larger sample sizes to strengthen precision. We do, however, report confidence intervals for the n=50 results in Appendix E.3-E4, which show, at the selected FanoutQA operating points, Qwen 3 (+14.7 95% CI [1.3, 28.0]) and Ministral 3 (+20.7 [8.7, 32.7]) and an average gain across model families of (+13.5 [6.5, 20.3]; Table 8).
Moreover, across the full FanoutQA results, CacheBack achieves a higher mean strict accuracy and lower median TTEOA than its equivalent text worker at 4x, 8x, and 16x compression in all four model families (Tables 6- 7). The observed improvements are therefore shown across multiple compression ratios and model architectures.
As you mentioned, the Longbench V2 Easy choice is explained in §4.3 and Appendix F.3. However, easy remains challenging for the models of this size. This is shown by the fact that same-size text workers achieve 39% in both model families compared to chance at 25%.
Our baselines isolate the effect of receiver conditioning in communication. Text workers are given the same query; therefore, it is only in how the message is constructed (§4) and, where feasible, full context isolates selection (§4.2).
We agree that broader workflows would help strengthen application to agentic workflows. Our current benchmarks tests evidence aggregation across workers (FanOutQA) and preservation through successive handoffs (LongBench) (§4). We will discuss these evaluated workflows from the potential applications discussed in §5. If you would like we could run something closer to code based reasoning such as LOC Bench.