- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
KVise proposes a training-free, receiver-conditioned KV-cache selection method for distributed LLM collaboration. It estimates the utility of remote KV shards using receiver attention queries and mergeable online-softmax statistics, then transfers selected shards for joint attention.
Across four model families and four multi-hop QA datasets, the paper reports 30.2–61.2% reductions in logical wire traffic and approximately 21% lower estimated latency compared with exhaustive KV transfer. Latency results are obtained through calibrated trace replay rather than measured end-to-end distributed execution.
Critical strengths
S1. Technically interesting receiver-conditioned KV selection.
Using native attention queries and mergeable online-softmax statistics to estimate remote KV utility before transmission is a technically interesting approach. It requires no additional model training and provides a principled mechanism for allocating communication according to receiver demand.
S2. Empirical evidence of communication reduction.
The evaluation spans four model families and four QA datasets. The reported 30.2–61.2% logical wire reduction demonstrates that substantial KV traffic can be eliminated relative to exhaustive transfer. The distortion-threshold analysis also provides useful evidence of a controllable communication–quality trade-off, including a setting where communication decreases without an observed F1 reduction.
Critical weaknesses
W1. The primary latency improvements are estimated rather than measured in a distributed system.
The paper reports approximately 21% improvement in TTFT and completion latency. However, these results come from calibrated trace replay under an assumed 10 Mbps network setting, rather than actual end-to-end measurements across distributed hosts.
The experiments use PyTorch/Transformers on a single server with four RTX A6000 GPUs. Although the paper analyzes logical control-plane traffic, it does not sufficiently establish the wall-clock overhead of receiver probing, sender-side scoring, shard selection, network communication, and suffix replay in a real distributed deployment.
This is a fundamental limitation for a paper claiming efficient distributed inference. Reducing logical communication bytes does not necessarily translate into equivalent latency improvements once additional computation, synchronization, and communication rounds are included.
W2. Communication savings are accompanied by substantial quality degradation in several settings.
Table 2 shows that KVise reduces communication volume, but downstream task quality frequently decreases.
For example:
- Phi-4 on 2WikiMQA: F1 decreases from 0.2442 to 0.1567.
- Qwen3-8B on HotpotQA: F1 decreases from 0.5681 to 0.4879.
- Phi-4 on HotpotQA: F1 decreases from 0.3915 to 0.3174.
Quality point estimates decrease in 10 of the 16 model–dataset combinations. However, Figure 3 shows that the 95% confidence intervals for quality differences overlap zero in 11 of the 16 settings. Thus, while substantial degradation is observed in several settings, the evidence does not establish statistically significant degradation across all 10 cases.
The paper provides a useful quality–communication trade-off analysis for Llama-3.1-8B on HotpotQA, where quality can be maintained at a particular threshold. However, this does not establish comparable quality preservation across the other model–dataset combinations.
Consequently, the reported communication and estimated latency improvements cannot be interpreted as broadly effective efficiency gains at matched task quality. In several settings, KVise exchanges lower task quality for reduced communication.
W3. The incremental contribution over prior methods and practical alternatives is insufficiently established.
KVise's main algorithmic contribution is receiver-conditioned KV-shard selection using attention statistics. However, the paper does not adequately isolate the benefit of this mechanism from the more general benefit of transmitting fewer KV states.
KVComm (ICLR 2026) already investigates selective inter-agent KV sharing. Although the paper discusses KVComm qualitatively, it provides no quantitative comparison. Simpler policies, including random, static, and sender-side selection, are also absent. Without these baselines, it is difficult to determine whether receiver-conditioned selection provides a meaningful advantage over existing or simpler methods.
A second concern is the choice of communication representation. The evaluation does not establish when transferring KV states is preferable to transmitting text or tokens followed by receiver-side prefill. These alternatives do not preserve identical internal states and may differ in task quality, but they avoid the complexity of exchanging and aligning remote KV tensors. Comparing their end-to-end costs and task quality is important for establishing the practical value of the proposed approach.
Thus, the current evidence supports selective communication relative to exhaustive KV transfer, but does not sufficiently establish the incremental value of KVise's specific mechanism or its advantage over practical communication alternatives.
Overall assessment.
KVise presents an interesting receiver-conditioned selection mechanism and demonstrates meaningful reductions in logical KV traffic. Nevertheless, three central concerns remain: its headline latency improvements lack direct distributed-system validation, several settings exhibit substantial downstream quality degradation, and the incremental advantage over prior and simpler approaches is unclear.
These limitations directly affect the paper's main claims of efficient distributed LLM collaboration. In its current form, I do not find the evidence sufficient to support acceptance.
Questions for authors
Q1. Actual distributed performance.
Can the authors provide measured end-to-end TTFT and completion latency across physically separate hosts, including probing, scoring, selection, communication, and replay overheads?
Measured improvements that remain substantial after accounting for all these costs, and that agree reasonably with the trace-replay estimates, would materially strengthen the systems-efficiency claim underlying W1.
Q2. Quality-preserving efficiency.
For the model–dataset combinations exhibiting substantial F1 degradation, can the authors demonstrate meaningful communication and latency savings at matched task quality relative to exhaustive transfer?
Evidence that the reported efficiency gains persist across multiple models and tasks without substantial quality regressions would directly address W2.
Q3. Incremental contribution and practical alternatives.
Can the authors compare KVise with KVComm, simple KV-selection policies, and text/token transmission followed by receiver-side prefill under comparable workloads?
A consistent advantage in quality-preserving communication efficiency or end-to-end latency would help establish that receiver-conditioned KV selection provides value beyond existing and simpler alternatives, addressing W3.
Q4. Scalability of the selection mechanism.
How does selection overhead scale with the number of senders, KV shards, and context length? In particular, does the additional computation required for receiver-conditioned scoring remain small enough to justify the resulting communication savings?
Measured scaling results demonstrating that selection overhead remains manageable under larger workloads would further strengthen the practical applicability of KVise.