acceptodds
Withdrawn from ICLR 2027

KVISE: selective KV sharing for efficient distributed LLM collaboration

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Communication overhead can become a major bottleneck in collaborative LLM inference when agents exchange large intermediate states. Although KV-cache sharing enables direct reuse of sender-side computation, exhaustively transferring remote caches is often inefficient because KV utility is highly non-uniform and depends on the receiver's current computation. We present KVise, a training-free and plug-and-play framework that formulates inter-agent KV exchange as receiver-conditioned communication allocation. The receiver provides compact native attention queries, while senders summarize candidate KV shards using mergeable online-softmax statistics, enabling their utility to be estimated before transmitting raw tensors. KVise then selects shards according to their marginal attention-distortion reduction per byte and jointly reuses the selected states with the receiver's local cache. Across four model families and four multi-hop QA datasets, KVise reduces logical wire traffic by 30.2–61.2% and mean TTFT and completion latency by approximately 21% under our main network setting. Further analyses show concentrated receiver-conditioned KV utility, a controllable quality–communication trade-off, and diminishing latency gains as communication becomes less dominant. These results demonstrate that selective, receiver-aware KV transfer can substantially reduce unnecessary inter-agent communication while preserving flexible control over efficiency and task quality.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Large Language Models, KV Cache, Multi-Agent Systems, Communication-Efficient Inference, Attention-Aware KV Selection

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 4 reviews, above 9% of the others.
Reviewer_9AauOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

KVise proposes a training-free, receiver-conditioned KV-cache selection method for distributed LLM collaboration. It estimates the utility of remote KV shards using receiver attention queries and mergeable online-softmax statistics, then transfers selected shards for joint attention.

Across four model families and four multi-hop QA datasets, the paper reports 30.2–61.2% reductions in logical wire traffic and approximately 21% lower estimated latency compared with exhaustive KV transfer. Latency results are obtained through calibrated trace replay rather than measured end-to-end distributed execution.

Critical strengths

S1. Technically interesting receiver-conditioned KV selection.

Using native attention queries and mergeable online-softmax statistics to estimate remote KV utility before transmission is a technically interesting approach. It requires no additional model training and provides a principled mechanism for allocating communication according to receiver demand.

S2. Empirical evidence of communication reduction.

The evaluation spans four model families and four QA datasets. The reported 30.2–61.2% logical wire reduction demonstrates that substantial KV traffic can be eliminated relative to exhaustive transfer. The distortion-threshold analysis also provides useful evidence of a controllable communication–quality trade-off, including a setting where communication decreases without an observed F1 reduction.

Critical weaknesses

W1. The primary latency improvements are estimated rather than measured in a distributed system.

The paper reports approximately 21% improvement in TTFT and completion latency. However, these results come from calibrated trace replay under an assumed 10 Mbps network setting, rather than actual end-to-end measurements across distributed hosts.

The experiments use PyTorch/Transformers on a single server with four RTX A6000 GPUs. Although the paper analyzes logical control-plane traffic, it does not sufficiently establish the wall-clock overhead of receiver probing, sender-side scoring, shard selection, network communication, and suffix replay in a real distributed deployment.

This is a fundamental limitation for a paper claiming efficient distributed inference. Reducing logical communication bytes does not necessarily translate into equivalent latency improvements once additional computation, synchronization, and communication rounds are included.

W2. Communication savings are accompanied by substantial quality degradation in several settings.

Table 2 shows that KVise reduces communication volume, but downstream task quality frequently decreases.

For example:

  • Phi-4 on 2WikiMQA: F1 decreases from 0.2442 to 0.1567.
  • Qwen3-8B on HotpotQA: F1 decreases from 0.5681 to 0.4879.
  • Phi-4 on HotpotQA: F1 decreases from 0.3915 to 0.3174.

Quality point estimates decrease in 10 of the 16 model–dataset combinations. However, Figure 3 shows that the 95% confidence intervals for quality differences overlap zero in 11 of the 16 settings. Thus, while substantial degradation is observed in several settings, the evidence does not establish statistically significant degradation across all 10 cases.

The paper provides a useful quality–communication trade-off analysis for Llama-3.1-8B on HotpotQA, where quality can be maintained at a particular threshold. However, this does not establish comparable quality preservation across the other model–dataset combinations.

Consequently, the reported communication and estimated latency improvements cannot be interpreted as broadly effective efficiency gains at matched task quality. In several settings, KVise exchanges lower task quality for reduced communication.

W3. The incremental contribution over prior methods and practical alternatives is insufficiently established.

KVise's main algorithmic contribution is receiver-conditioned KV-shard selection using attention statistics. However, the paper does not adequately isolate the benefit of this mechanism from the more general benefit of transmitting fewer KV states.

KVComm (ICLR 2026) already investigates selective inter-agent KV sharing. Although the paper discusses KVComm qualitatively, it provides no quantitative comparison. Simpler policies, including random, static, and sender-side selection, are also absent. Without these baselines, it is difficult to determine whether receiver-conditioned selection provides a meaningful advantage over existing or simpler methods.

A second concern is the choice of communication representation. The evaluation does not establish when transferring KV states is preferable to transmitting text or tokens followed by receiver-side prefill. These alternatives do not preserve identical internal states and may differ in task quality, but they avoid the complexity of exchanging and aligning remote KV tensors. Comparing their end-to-end costs and task quality is important for establishing the practical value of the proposed approach.

Thus, the current evidence supports selective communication relative to exhaustive KV transfer, but does not sufficiently establish the incremental value of KVise's specific mechanism or its advantage over practical communication alternatives.

Overall assessment.

KVise presents an interesting receiver-conditioned selection mechanism and demonstrates meaningful reductions in logical KV traffic. Nevertheless, three central concerns remain: its headline latency improvements lack direct distributed-system validation, several settings exhibit substantial downstream quality degradation, and the incremental advantage over prior and simpler approaches is unclear.

These limitations directly affect the paper's main claims of efficient distributed LLM collaboration. In its current form, I do not find the evidence sufficient to support acceptance.

Questions for authors

Q1. Actual distributed performance.

Can the authors provide measured end-to-end TTFT and completion latency across physically separate hosts, including probing, scoring, selection, communication, and replay overheads?

Measured improvements that remain substantial after accounting for all these costs, and that agree reasonably with the trace-replay estimates, would materially strengthen the systems-efficiency claim underlying W1.

Q2. Quality-preserving efficiency.

For the model–dataset combinations exhibiting substantial F1 degradation, can the authors demonstrate meaningful communication and latency savings at matched task quality relative to exhaustive transfer?

Evidence that the reported efficiency gains persist across multiple models and tasks without substantial quality regressions would directly address W2.

Q3. Incremental contribution and practical alternatives.

Can the authors compare KVise with KVComm, simple KV-selection policies, and text/token transmission followed by receiver-side prefill under comparable workloads?

A consistent advantage in quality-preserving communication efficiency or end-to-end latency would help establish that receiver-conditioned KV selection provides value beyond existing and simpler alternatives, addressing W3.

Q4. Scalability of the selection mechanism.

How does selection overhead scale with the number of senders, KV shards, and context length? In particular, does the additional computation required for receiver-conditioned scoring remain small enough to justify the resulting communication savings?

Measured scaling results demonstrating that selection overhead remains manageable under larger workloads would further strengthen the practical applicability of KVise.

Reviewer_jZkLOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes KVise, a training-free framework for selective KV-cache communication between compatible LLM agents. The receiver sends query probes, and senders return compact attention statistics for candidate KV shards. The receiver then selects shards based on their marginal attention-distortion reduction per transmitted byte. Experiments show reduced communication and estimated latency compared with full-KV transfer, with task-dependent quality changes.

Critical strengths
  1. The separation of lightweight utility estimation from expensive KV transfer is sensible.
  2. Mergeable attention statistics enable subset evaluation before transmitting raw KV tensors.
  3. The selection rule accounts for both communication cost and redundancy among selected shards.
  4. Evaluation spans multiple model families and datasets and examines bandwidth and quality trade-offs.
Critical weaknesses
  1. Missing text-transfer baseline. Under the low-bandwidth conditions where KVise benefits most, transmitting agent outputs as text and re-prefilling them at the receiver may be substantially cheaper. Comparing only against full-KV transfer does not establish the practical advantage of KV communication.

  2. Unmeasured quality loss from KV reuse. Re-RoPE corrects positional encoding but does not resolve contextual differences between sender-produced KV states and receiver-side recomputation. The experiments measure additional losses from selection without isolating losses from cross-context reuse.

  3. Limited latency evidence. Latency is estimated through calibrated trace replay. Gains diminish substantially as bandwidth increases, and decoding-specific acceleration is not measured separately. Insufficient selection baselines and ablations. Comparisons with random selection, attention top-k, and alternative probe or shard configurations are missing.

  4. Limited architectural coverage. Hybrid attention architectures may have much smaller KV caches, potentially reducing transfer savings enough that probing and selection overhead outweigh the benefits.

Questions for authors
  1. How does KVise compare with transmitting the same sender-generated text and re-prefilling it at the receiver, in both end-to-end latency and answer quality?

  2. How much quality loss comes from full-KV reuse itself, separately from selective KV removal?

  3. What are the absolute costs of probing, computing statistics, selecting shards, transferring KV, and replaying the receiver suffix?

  4. At what bandwidth and context length does KVise become faster than text transfer followed by re-prefill?

  5. Does KVise reduce time per output token, and how does it compare with receiver-side sparse attention or KV compression after re-prefill?

  6. Do the benefits persist with hybrid attention architectures?

  7. How are probe layers and heads selected, and do experimental shards span all layers or specific layer groups?

Reviewer_XapuOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

KVise selectively transfers remote KV states for distributed LLM collaboration. Its claimed contribution is receiver-conditioned, pre-transfer subset evaluation using mergeable attention statistics, followed by selection based on distortion reduction per byte. The results support communication savings over exhaustive transfer, particularly under limited bandwidth. However, novelty remains insufficiently established, and the lack of competitive baselines and ablations makes it difficult to assess the selector’s specific value. Substantial quality drops in some settings also weaken the efficiency claims.

Critical strengths
  • The paper addresses a practical communication bottleneck in distributed LLM inference.
  • The framework requires no additional training and is compatible with KV compression.
  • The experiments show substantial communication savings over exhaustive transfer.
Critical weaknesses
  1. The experiments do not establish the advantage of the proposed selection mechanism. Table 2 compares KVise only with exhaustive transfer. It includes neither simple selection heuristics nor KVComm, despite discussing the latter as related work. The results therefore demonstrate the benefit of transferring fewer KV states, but do not establish that this particular selector is necessary.

  2. Some communication savings come with substantial quality degradation. In Table 2, Phi-4’s F1 on 2WikiMQA drops from 0.2442 to 0.1567, while Qwen’s HotpotQA F1 drops from 0.5681 to 0.4879. These results make aggregate communication and latency reductions insufficient to establish a favorable quality–efficiency trade-off.

  3. The experiments do not isolate the contributions of the core components. Sections 3.2–3.4 examine the distortion threshold, network bandwidth, and utility concentration, but do not ablate receiver-conditioned probes, subset-dependent scoring, or byte-normalized selection. Consequently, the individual value of these design choices remains unclear.

  4. The evidence for reliable quality control is narrow. The threshold sweep in Table 3 covers only Llama-3.1-8B on HotpotQA. At τ=0.02\tau=0.02, its F1 remains close to exhaustive transfer, whereas Table 2 shows substantial degradation for other model–dataset pairs under the same default threshold. A single threshold does not provide consistent quality preservation across settings.

  5. The latency evidence is limited to estimated performance and shows diminishing benefits. Section 3.1 uses calibrated trace replay rather than direct distributed latency measurements. Table 4 further shows that, at 20 Mbps, mean completion-latency reduction falls to 3.94%, while p95 completion latency worsens by 8.37%. The demonstrated speedup is therefore strongly dependent on network conditions.

  6. The novelty claim requires a sharper distinction from prior work. The related-work section already identifies query-dependent KV selection and selective inter-agent KV sharing. The proposed framework combines these directions with mergeable online-softmax statistics and benefit-per-byte selection, but the current comparisons do not establish how much this combination advances beyond those existing approaches.

Questions for authors

see the weakness

Reviewer_tA7mOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes KVise, a training-free framework for selective KV sharing between compatible LLM agents. The receiver provides provisional attention queries, and senders return compact, mergeable online-softmax statistics that allow candidate KV shards to be evaluated before transmitting their payloads. The receiver selects a subset according to an attention-distortion criterion and reuses the transferred states through positional alignment and joint attention. Across four model families and four multi-hop QA datasets, the paper reports a 30.19–61.16% reduction in logical wire traffic. The reported average TTFT and completion-latency reductions of approximately 21% are based on calibrated trace-replay estimates. The central contribution is receiver-conditioned communication allocation before KV transfer.

Critical strengths
  1. The pre-transfer allocation problem is well motivated, and the proposed mechanism directly addresses it. Using compact attention statistics to estimate the utility of remote KV shards avoids requiring the receiver to obtain the full payload before deciding what to retain. The mergeability of the softmax statistics provides a principled way to reconstruct attention responses for fixed probes over candidate shard unions. This is a useful contribution to communication-efficient LLM collaboration.

  2. The evaluation demonstrates substantial communication savings and exposes meaningful operating trade-offs. The 16 model–dataset settings provide evidence that selective transfer can substantially reduce logical traffic. The threshold and bandwidth studies also help identify where the approach is useful, including settings with diminishing latency benefits and degraded tail latency. These results provide useful information beyond a single favorable configuration.

Critical weaknesses
  1. The practical latency claim is not sufficiently validated against measured distributed execution. The TTFT and completion-latency results in Table 2 are calibrated trace-replay estimates, but the paper does not report an independent measured-versus-predicted validation for the complete distributed protocol. Consequently, the reported reductions establish benefits under the replay model without establishing how accurately those benefits translate to deployment. This distinction matters because the protocol involves receiver probing, sender-side evaluation, control communication, subset selection, KV transfer, and receiver suffix replay. At 20 Mbps, Table 4 reports only a 3.94% estimated mean completion-latency reduction, while p95 TTFT and completion latency worsen by 9.16% and 8.37%, respectively. With such a small benefit margin, validation of the complete critical path is important to the overall assessment.

  2. The reported communication savings sometimes come with substantial task-quality losses, and the default operating point is not adequately justified across settings. Table 2 contains lower quality point estimates in 10 of the 16 settings. For example, Phi-4 on 2WikiMQA drops from 0.2442 to 0.1567 token F1, a 35.8% relative decrease; Qwen3-8B on HotpotQA drops from 0.5681 to 0.4879, a 14.1% relative decrease. These point-estimate differences should not all be interpreted as statistically significant, but their magnitude makes the quality–communication trade-off consequential. The favorable threshold sweep on Llama/HotpotQA in Table 3 does not establish a reliable operating policy for the other model–task pairs. Low distortion on provisional attention probes does not itself ensure preservation of downstream answer quality. The current evidence therefore leaves unclear how the method should be configured when quality preservation is a deployment requirement.

Questions for authors
  1. How was the trace-replay model calibrated, and what is its prediction error against measured end-to-end distributed runs? A limited validation on representative configurations, covering both communication-dominated and lower-benefit settings, would materially affect my assessment. Please clarify how the complete probe/control, sender evaluation, selection, transfer, and receiver replay costs enter the timing model.

  2. How should the distortion threshold be selected for a new model–task pair with a specified quality-loss tolerance? Please clarify the calibration/evaluation separation and whether the larger losses in Table 2 persist at a quality-preserving operating point. The associated communication and latency trade-offs would help determine whether the practical benefits remain meaningful.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.