acceptodds
Under review as a conference paper at ICLR 2027

KV-Pipe: On the Relation Between KV Sharing and Pipeline Parallel Efficiency in LLMs

Abstract

Pipeline parallelism (PP) is bounded by its slowest stage. In LLMs, however, the stages are not identical: the last stage also carries the LM head, so an evenly split decoder already leaves every other device idle for part of each step. Moving whole layers between stages cannot fix this, because a single layer is coarser than the imbalance itself. We observe that cross-layer KV sharing, which has so far been studied as an inference-time memory optimization, is exactly the fine-grained knob this problem needs. Each shared layer removes a small, predictable amount of compute from its stage. We present , which places shared-KV layers on the bottleneck stage and chooses how many to share by minimizing a FLOPs Imbalance Ratio (FIR) computed offline from per-layer FLOP counts. For LLaMA2-7B with PP8 on Ascend 910B NPUs, sharing KV in only four layers removes 1.5% of model FLOPs but cuts iteration time by 9.8% and raises MFU from 56.3% to 61.5%. Across the sweep, FIR orders measured MFU almost perfectly (Spearman at PP8). Sharing more layers makes both FIR and MFU worse again, even though total FLOPs keep falling. When we vary the LM-head cost, the budget predicted from FIR matches the measured optimum in all five settings. The gains grow with pipeline depth, also in a controlled sweep that fixes everything but the number of stages. On LLaMA2-7B and LLaMA3-8B, validation perplexity and downstream accuracy stay within the run-to-run variation of full attention. The gains are additive to the Seq1F1B schedule, carry over to LLaMA2-13B, LLaMA3-8B and Qwen2.5-14B on GPUs, and the same kind of late-layer sharing speeds up GQA decoding by 7–8%.

Primary area:
foundation or frontier models, including LLMs
Keywords:
KV Sharing, Pipeline Parallel, Efficient LLMs

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_K24nOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work introduces KVPipe, a technique to reduce load imbalance between worker nodes in pipeline parallel scenarios. The main point is that because of the non-uniformity of transformer computations across layers (last layer involving the LM head which has significant load), the work that each slice of the pipeline must do it not equally split. Since the slices must shuffle batches in lockstep, nodes with less work will always be underutilized. KVPipe fixes this problem for scenarios where the architecture of the model can be changed; specifically, if we can allow KV-sharing between select layers, we can select the layers that skip KV computation by reusing prior layers' KV with the intention of rebalancing the load. KVPipe is tested on several models in the LLama and Qwen model families on Ascend GPUs and Nvidia V100 GPUs, showing 2-10% improvement in iteration time.

Critical strengths

I'd like to thank the authors for submitting their work. The problem setup of load imbalance in pipeline parallelism is an interesting problem. KVPipe's solution is able to achieve good balance.

Critical weaknesses

Unfortunately, I do not believe the paper has substantial innovation:

  • A major limitation of KVPipe is that it ties the model's architecture to a very specific pipeline parallelism setup. For example, if the model was designed to be optimal for PP=8 during training, the last 4 layers are made into shared KV layers. However, if at serving time we use PP=4, the optimal would have been 6 layers. At serving time we cannot rearchitect the model and must continue to use the model in an unbalanced way. In other words, KVPipe creates model architectures that are tied to a specific GPU and PP config, and will not be efficient in other settings.
  • The main insight seems to be that if there are multiple sources of imbalance (KV-sharing and LM head), if we can control one source we can cancel out the effects of the other. While I do agree that this is a matter of taste, I do not find this insight to be sufficiently novel.
  • Based on the results in §4.4, it is not clear what the empirical improvement over echo-style selection is.
Questions for authors

Please see weaknesses.

2 comments

We thank the reviewer for the constructive feedback. To summarize, C1 addresses architecture portability across PP configurations and the distinction between analytical and runtime optimality; C2 clarifies KV-Pipe's novelty in sub-layer architectural balancing and predictive budget selection; and C3 explains its advantages over ECHO-style sharing in terms of comparable throughput, fewer architectural changes, and quality preservation. We hope these responses address your concerns and would appreciate your reconsideration of our work and its evaluation.

C1: We thank the reviewer for raising this practical concern. KV-Pipe's trained architecture is portable across PP degrees, although its runtime-optimal sharing budget may vary. Proposition 1 shows that, under the uniform-saving, single-bottleneck assumptions, the FIR-optimal budget m⋆≈H/Δm^\star\approx H/\Delta is independent of PP degree. For LLaMA2-7B at 8K, m⋆=4m^\star=4 at both PP=4 and PP=8. The six-layer configuration at PP=4 results from optional runtime refinement rather than a different analytical optimum. Our PP=4 experiments provide supporting evidence: the four-layer architecture selected for PP=8 achieves 63.41% MFU, compared with 61.28% for full attention and 63.97% for the refined six-layer configuration, recovering approximately 80% of the best measured MFU improvement. Importantly, these results compare sharing architectures and do not establish performance of the identical PP=8-trained checkpoint during PP=4 serving. Once trained, the KV-sharing architecture remains fixed, but the checkpoint can be deployed with different pipeline partitions without architectural modification. We acknowledge that the optimal runtime balance may change across PP degrees, hardware, and serving workloads. Nevertheless, the reduction in KV-cache memory and K/V projections persists independently of the pipeline configuration.

C2. One clarification is that KV sharing is not a source of the original imbalance, but a fine-grained mechanism to mitigate it. The imbalance arises primarily from the LM-head computation on the final pipeline stage. While load balancing is well established, KV-Pipe contributes in three ways:

  1. Sub-layer architectural balancing. Existing pipeline methods typically assume fixed per-layer costs. At PP=8, even partitioning is already optimal at the whole-layer level, yet the LM head creates imbalance. To our knowledge, KV-Pipe is the first to use cross-layer KV sharing specifically as a fine-grained stage-cost control.

  2. Predictive placement and budget selection. Under the single-bottleneck, uniform-saving assumptions, we derive m⋆≈H/Δm^\star\approx H/\Delta, which matches the measured optimum in all five controlled LM-head-surplus experiments. Equal-FLOP placements also yield substantially different iteration times: 257.1 s (Uniform), 251.2 s (Symmetric-Bipolar), and 237.5 s (KV-Pipe).

  3. Efficiency with fewer modifications. At PP=8, converting four layers removes only 1.5% of model FLOPs but reduces iteration time by 9.8%. Compared with fixed 25% sharing, KV-Pipe achieves nearly identical throughput with half as many converted layers.

C3. KV-Pipe's advantage over ECHO-style fixed 25% sharing is not higher peak throughput, but achieving comparable performance with fewer architectural changes through cost-aware budget selection. In LLaMA2-7B, both approaches achieve nearly identical iteration times at PP=8 (−9.8% vs. ~−10.0%), while fixed-ratio sharing is slightly faster at PP=4 (−7.0% vs. −6.3%). At PP=2, the configurations coincide. We will clarify these results in §4.4.

The key advantages of KV-Pipe are: Fewer conversions for comparable speed. At PP=8, four shared layers achieve 237.5 s, versus ~237.0 s with eight layers. Additional conversions provide negligible runtime benefit, so KV-Pipe achieves essentially the same speed with half as many architectural changes. Better quality preservation. LLaMA2-7B quality differences are within observed run variability, so we do not claim a significant quality improvement there. However, the three-run LLaMA3-8B study shows that four-layer sharing maintains PPL close to full attention (6.23±0.026.23\pm0.02 vs. 6.21±0.026.21\pm0.02), whereas fixed 25% sharing increases it to 6.28±0.036.28\pm0.03. This supports limiting unnecessary conversions, although the four-layer budget was not independently optimized for LLaMA3-8B. Adaptation to pipeline imbalance. A fixed 25% ratio ignores stage costs and may over- or under-correct the bottleneck. In our controlled LM-head-surplus experiments, the optimal budget varies from two to six layers, and FIR predicts the measured optimum in all five settings. A single fixed sharing ratio cannot adapt to these changes.

Dear Reviewer,

We thank you again for the constructive feedback. To summarize, C1 addresses architecture portability across PP configurations and the distinction between analytical and runtime optimality; C2 clarifies KV-Pipe's novelty in sub-layer architectural balancing and predictive budget selection; and C3 explains its advantages over ECHO-style sharing in terms of comparable throughput, fewer architectural changes, and quality preservation.

We hope these responses address your concerns and would appreciate your reconsideration of our work and its evaluation.

Reviewer_YLuwOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper treats cross-layer KV sharing as a sub-layer cost knob for pipeline parallelism, and defines a FLOPs Imbalance Ratio (FIR, maximum stage FLOPs over the mean), shows under a single-heavy-stage model that the balance-optimal budget is H/Delta and independent of the pipeline degree, and places conversions with a greedy deepest-first loop.

Critical strengths
  1. The arithmetic is exact and the idea is interesting.

  2. The ablation is sufficient and the theoretical analysis is clear.

Critical weaknesses
  1. The central causal claim is close to entailed by how the metric is defined.

  2. The quality claim cannot be checked. The generality claim rests on models where the paper's own accounting says the mechanism cannot operate.

Questions for authors
  1. Can you report wall clock with repeats at PP=8 for the fixed 25 percent tail policy and for a cost-aware non-uniform partition, and compare on time rather than on MFU?

  2. What is the shared-KV budget, converted-layer list and FIR for every cell of Tables 7, 15 and 16, and how do the GQA gains arise given H/Delta of 63 and 74?

  3. How is tau(l) defined, and how do the source keys and values reach stage 7 at PP=8?

2 comments

We thank the reviewer for the constructive feedback. To summarize, Critical Weakness 1 (FIR–MFU correlation and causal evidence) is addressed in P1, while Critical Weakness 2 (quality, reproducibility, and GQA generality) is addressed in P2 and P4. Regarding the specific questions, Q1 (wall-clock comparisons) is addressed in P3, Q2 (sharing configurations and GQA gains) in P4, and Q3 (KV-source mapping and communication) in P5. We hope our responses address your concerns and would appreciate your reconsideration of our work and its evaluation.

P1. Under the idealized steady-state model, MFU is proportional to 1/FIR1/\mathrm{FIR}; hence their strong correlation is mathematically expected, not independent causal evidence. MFU remains our principal efficiency metric, while FIR is the inexpensive offline placement objective and measured iteration time establishes practical throughput benefits.

At a fixed budget of four shared layers, Uniform, Symmetric-Bipolar, and bottleneck-first placement all require 4405 TFLOPs per iteration, yet take approximately 257.1, 251.2, and 237.5 s, respectively. Increasing the budget from four to sixteen removes another 4.6% of FLOPs without materially improving iteration time (237.5 versus approximately 237.3 s). These equal-work and oversharing comparisons—not the FIR–MFU correlation—support the claim that where work is removed matters to wall-clock performance.

P2 (Quality and reproducibility). Our shared-KV models are adapted from pretrained checkpoints rather than trained from scratch. Please consider our response to Reviewer Hci1.

For LLaMA3-8B, Table 6 reports three runs per variant: full attention obtains 6.21±0.026.21\pm0.02 PPL, four late-layer conversions 6.23±0.026.23\pm0.02, and fixed 25% sharing (eight layers) 6.28±0.036.28\pm0.03; downstream averages are 60.6, 60.5, and 60.3, respectively. The LLaMA2-7B results currently reflect individual PP-specific runs, and we will distinguish them from multi-seed comparisons and clarify checkpoint matching.

P3. Let aa denote the FLOP cost of four LLaMA2-7B decoder layers. The LM head contributes approximately 0.139a0.139a, so an even 32-layer/8-stage split has maximum stage cost 1.139a1.139a. Under the paper's identical-layer FLOP model, any non-uniform whole-layer partition necessarily places at least five layers on some stage, yielding a cost of at least 1.25a1.25a. Thus, the even split is already the best whole-layer partition under this model; we will also measure an explicit uneven partition, [4,4,4,4,4,4,5,3][4,4,4,4,4,4,5,3].

The existing PP=8 measurements show approximately 237.0 s for fixed 25% tail sharing (layers 25–32) and 237.5 s for KV-Pipe (layers 29–32): a ~0.2% difference. KV-Pipe reaches essentially the same time with half as many converted layers, limiting architectural change. The separate LLaMA3-8B quality results above support preferring a smaller sharing budget when throughput is comparable. We will report repeated wall-clock measurements for the partition and sharing baselines.

P4. For the FLOP-based model, H/Δ≈3.9H/\Delta\approx3.9 for LLaMA2-7B and ≈3.1\approx3.1 for LLaMA2-13B, versus ≈62.6\approx62.6 for LLaMA3-8B and ≈74.2\approx74.2 for Qwen2.5-14B. At PP=8 and 8K, even converting every layer on the last stage predicts only approximately 2.0% and 2.1% reductions in the bottleneck compute for the two GQA models, whereas the reported iteration-time reductions are 7.2% and 8.9%. The LLaMA2-13B PP=2/4 results also exceed their idealized predictions. We therefore do not treat these gains as quantitative validation of the present FIR cost model. We will separate them as empirical observations, reconcile the modified-model FLOP/MFU accounting, and use forward/backward stage-time, kernel, communication, and pipeline-idle profiles before assigning a causal mechanism.

As requested, we will supply a per-configuration table for Tables 7, 15, and 16 with the verified sharing budget, converted-layer indices, source mapping, stage partition, and computed FIR. For Table 16, measured stage-time imbalance will remain distinct from FLOP-based FIR. The uniform-Δ\Delta optimality result described to Reviewer 1 concerns the FIR allocation objective, not a guarantee of optimal wall-clock time.

P5. We use the effective mapping τ(ℓ)=max⁡{j<ℓ:zj=0}\tau(\ell)=\max\{j<\ell:z_j=0\}: each converted layer reuses the nearest preceding unconverted attention layer's K/V. At PP=8 with four conversions, stage 6's layer 28 supplies (K28,V28)(K_{28},V_{28}) to stage 7's layers 29–32. As detailed in our response to Reviewer Hci1 (P1–P3), stage 6 asynchronously sends one shared K/V pair per microbatch via dedicated P2P; stage 7 reuses it across four layers and returns accumulated K/V gradients once. At 8K/BF16/MBS=1, this adds 128 MiB per direction and 0.125 GiB per in-flight microbatch on stage 7. We will report exposed latency and buffer accounting. PP=2/4 use local sources; larger budgets may cross more boundaries.

Dear Reviewer,

We thank you again for the constructive feedback. To summarize, Critical Weakness 1 (FIR–MFU correlation and causal evidence) is addressed in P1, while Critical Weakness 2 (quality, reproducibility, and GQA generality) is addressed in P2 and P4. Regarding the specific questions, Q1 (wall-clock comparisons) is addressed in P3, Q2 (sharing configurations and GQA gains) in P4, and Q3 (KV-source mapping and communication) in P5.

We hope our responses address your concerns and would appreciate your reconsideration of our work and its evaluation.

Reviewer_Hci1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes KV-Pipe, a stage-aware cross-layer key-value sharing approach for improving load balance in pipeline-parallel LLM training. The authors provide a simplified analytical model for the optimal sharing budget and evaluate the approach on Ascend NPUs and NVIDIA GPUs. On LLaMA2-7B with eight pipeline stages, KV-Pipe reduces iteration time by 9.8% while removing approximately 1.5% of the model FLOPs.

Critical strengths
  1. The paper provides a useful perspective on cross-layer KV sharing as a fine-grained mechanism for adjusting stage-level computational costs, rather than solely reducing KV-cache memory or attention computation.
  2. The experiments on llama2-7B demonstrate that a small reduction in total FLOPs can lead to a substantially larger reduction in iteration time when the savings are concentrated on the bottleneck stage.
Critical weaknesses
  1. A central concern is how KV dependencies are handled across pipeline-stage boundaries. However, the paper does not specify the KV-source mapping, the corresponding communication mechanism, or its runtime and memory costs. Since FIR explicitly ignores communication and backward-pass asymmetries, this omission makes it difficult to establish how accurately the proposed cost model reflects the actual execution of KV-Pipe.
  2. The greedy procedure is not guaranteed to be globally optimal for heterogeneous stage costs, and the experiments do not exercise its bottleneck-retargeting behavior. Moreover, the GQA experiments present a gap between the analytical explanation and the reported performance.
Questions for authors
  1. In the PP=8 configuration, layers 29–32 all use shared KV representations. Which earlier layer supplies these representations, and does this introduce additional communication across the boundary between pipeline stages 6 and 7?
  2. What explains the performance improvements on GQA models?
  3. Is the optimization target MFU or actual training throughput?
  4. Since KV-Pipe changes the attention architecture, the quality-efficiency tradeoff is essential to assessing its practical value. Could the authors clarify whether the models are trained from scratch or adapted from pretrained checkpoints, and report the training-token budgets, initialization procedures, datasets, and seed-level variability?
2 comments

We thank the reviewer for the constructive feedback. To summarize, Critical Weakness 1 (cross-stage KV dependencies and communication costs) is addressed in P1–P3, while Critical Weakness 2 (greedy optimality, bottleneck retargeting, and GQA performance) is addressed in P4 and P9. Regarding the specific questions, Q1 is answered in P1–P3, Q2 in P9, Q3 in P10, and Q4 in P5–P8. We hope these responses address your main concerns and encourage you to raise your score.

P1. Each converted layer reuses the nearest preceding full-attention layer's K/V. For PP=2 and PP=4, the selected source and consumers reside on the same stage, requiring no additional communication. At PP=8 (SKV=4), layer 28 on stage 6 supplies K/V to layers 29–32 on stage 7.

P2. K/V is transmitted once per microbatch, asynchronously on a dedicated P2P stream immediately after projection, overlapping with the remaining source-layer computation. All four consumers reuse the received buffer; their backward K/V gradients are accumulated and returned once. At BF16, 8K context and MBS=1, the additional traffic is 128 MiB forward + 128 MiB backward. On NVSwitch-connected A100s, we estimate ~0.7 ms raw transfer per direction, with ~0.2 ms exposed forward latency under effective overlap.

P3. Our reported iteration-time improvements are based on end-to-end execution. FIR remains our inexpensive offline compute-balancing predictor. We will clarify communication-aware refinement using measured stage execution time plus unhidden communication, and document additional cross-stage dependencies for larger sharing budgets.

P4. Bottleneck-first allocation is globally optimal for uniform KV-projection savings, even with heterogeneous stage costs and uneven partitions. In our experiments, convertible attention layers share the same (h,r), hence (Delta=4bsh²/r). For fixed (k), the average stage FLOPs is placement-independent; minimizing FIR therefore reduces to minimizing the maximum stage load, which bottleneck-first allocation solves optimally (exchange proof to be added). We refine the stopping rule: FIR may temporarily increase when the bottleneck shifts. We will update Alg. 1 to evaluate all (k\leq m) and retain the minimum FIR, preserving (O(mP)) complexity and guaranteeing global optimality under uniform (\Delta). Exhaustive search over 600 heterogeneous stage-cost configurations confirms 100% optimality, compared with 76–90% for the submitted stopping rule. For nonuniform (Delta), the method remains near-optimal empirically (mean FIR gap 0.08%, maximum 0.92% across 600 additional cases). We will add the proof, simulations, and first-/middle-stage retargeting examples to the appendix. Existing fixed-placement hardware measurements remain unchanged.

P5. Our models are adapted from pretrained LLaMA2-7B and LLaMA3-8B checkpoints, rather than trained from scratch. Following ECHO-LLaMA (Dialameh et al., 2025). We adopt its incremental adaptation protocol: approximately 150 steps (~4M tokens) per converted layer, followed by ~4B tokens of full-model continued training using SlimPajama-627B.

P6. We evaluate validation perplexity and downstream accuracy on MMLU, HellaSwag, ARC-Challenge and TruthfulQA using a consistent evaluation protocol. LLaMA3-8B results already report three runs per variant: full attention achieves 6.21±0.02 PPL and 60.6% downstream accuracy, versus 6.23±0.02 and 60.5% with four shared layers. For LLaMA2-7B, the current tables report individual-run values; we will distinguish these from multi-seed estimates.

P7. Pipeline degree changes distributed execution, not the model architecture. The two LLaMA2 full-attention PPL values (5.47 and 5.67) came from separate PP-specific runs and should not be interpreted as an architectural effect. We will clarify checkpoint provenance, standardize the quality comparison around matched full-attention references, and retain PP-specific results for system efficiency.

P8. At PP=8, four late-layer conversions improve MFU by 9.2% and reduce iteration time by 9.8%, while preserving downstream accuracy (52.2% versus 52.2% for full attention). Unlike fixed-ratio sharing, KV-Pipe converts only the layers needed to balance the bottleneck, limiting unnecessary architectural changes.

P9. The reported gains are end-to-end measurements on top of Seq1F1B; for example, LLaMA3-8B at PP=8 and 8K reduces iteration time from 278 to 258 s (7.2%). Execution-level effects beyond analytical FLOPs, including kernel efficiency, memory traffic, and pipeline scheduling, may contribute. We will clarify this distinction and add GQA-specific stage-level profiling to separate the FLOP-predicted benefit from additional runtime effects.

P10. KV-Pipe primarily targets pipeline efficiency, using FIR as an inexpensive optimization objective and MFU as the main evaluation metric. Actual training throughput is a complementary measure of practical benefit

Dear Reviewer,

We thank you again for the constructive feedback. To summarize, Critical Weakness 1 (cross-stage KV dependencies and communication costs) is addressed in P1–P3, while Critical Weakness 2 (greedy optimality, bottleneck retargeting, and GQA performance) is addressed in P4 and P9. Regarding the specific questions, Q1 is answered in P1–P3, Q2 in P9, Q3 in P10, and Q4 in P5–P8.

We hope these responses address your main concerns and encourage you to raise your score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.