- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This work introduces KVPipe, a technique to reduce load imbalance between worker nodes in pipeline parallel scenarios. The main point is that because of the non-uniformity of transformer computations across layers (last layer involving the LM head which has significant load), the work that each slice of the pipeline must do it not equally split. Since the slices must shuffle batches in lockstep, nodes with less work will always be underutilized. KVPipe fixes this problem for scenarios where the architecture of the model can be changed; specifically, if we can allow KV-sharing between select layers, we can select the layers that skip KV computation by reusing prior layers' KV with the intention of rebalancing the load. KVPipe is tested on several models in the LLama and Qwen model families on Ascend GPUs and Nvidia V100 GPUs, showing 2-10% improvement in iteration time.
Critical strengths
I'd like to thank the authors for submitting their work. The problem setup of load imbalance in pipeline parallelism is an interesting problem. KVPipe's solution is able to achieve good balance.
Critical weaknesses
Unfortunately, I do not believe the paper has substantial innovation:
- A major limitation of KVPipe is that it ties the model's architecture to a very specific pipeline parallelism setup. For example, if the model was designed to be optimal for PP=8 during training, the last 4 layers are made into shared KV layers. However, if at serving time we use PP=4, the optimal would have been 6 layers. At serving time we cannot rearchitect the model and must continue to use the model in an unbalanced way. In other words, KVPipe creates model architectures that are tied to a specific GPU and PP config, and will not be efficient in other settings.
- The main insight seems to be that if there are multiple sources of imbalance (KV-sharing and LM head), if we can control one source we can cancel out the effects of the other. While I do agree that this is a matter of taste, I do not find this insight to be sufficiently novel.
- Based on the results in §4.4, it is not clear what the empirical improvement over echo-style selection is.
Questions for authors
Please see weaknesses.
2 comments
Dear Reviewer,
We thank you again for the constructive feedback. To summarize, C1 addresses architecture portability across PP configurations and the distinction between analytical and runtime optimality; C2 clarifies KV-Pipe's novelty in sub-layer architectural balancing and predictive budget selection; and C3 explains its advantages over ECHO-style sharing in terms of comparable throughput, fewer architectural changes, and quality preservation.
We hope these responses address your concerns and would appreciate your reconsideration of our work and its evaluation.
We thank the reviewer for the constructive feedback. To summarize, C1 addresses architecture portability across PP configurations and the distinction between analytical and runtime optimality; C2 clarifies KV-Pipe's novelty in sub-layer architectural balancing and predictive budget selection; and C3 explains its advantages over ECHO-style sharing in terms of comparable throughput, fewer architectural changes, and quality preservation. We hope these responses address your concerns and would appreciate your reconsideration of our work and its evaluation.
C1: We thank the reviewer for raising this practical concern. KV-Pipe's trained architecture is portable across PP degrees, although its runtime-optimal sharing budget may vary. Proposition 1 shows that, under the uniform-saving, single-bottleneck assumptions, the FIR-optimal budget m⋆≈H/Δ is independent of PP degree. For LLaMA2-7B at 8K, m⋆=4 at both PP=4 and PP=8. The six-layer configuration at PP=4 results from optional runtime refinement rather than a different analytical optimum. Our PP=4 experiments provide supporting evidence: the four-layer architecture selected for PP=8 achieves 63.41% MFU, compared with 61.28% for full attention and 63.97% for the refined six-layer configuration, recovering approximately 80% of the best measured MFU improvement. Importantly, these results compare sharing architectures and do not establish performance of the identical PP=8-trained checkpoint during PP=4 serving. Once trained, the KV-sharing architecture remains fixed, but the checkpoint can be deployed with different pipeline partitions without architectural modification. We acknowledge that the optimal runtime balance may change across PP degrees, hardware, and serving workloads. Nevertheless, the reduction in KV-cache memory and K/V projections persists independently of the pipeline configuration.
C2. One clarification is that KV sharing is not a source of the original imbalance, but a fine-grained mechanism to mitigate it. The imbalance arises primarily from the LM-head computation on the final pipeline stage. While load balancing is well established, KV-Pipe contributes in three ways:
Sub-layer architectural balancing. Existing pipeline methods typically assume fixed per-layer costs. At PP=8, even partitioning is already optimal at the whole-layer level, yet the LM head creates imbalance. To our knowledge, KV-Pipe is the first to use cross-layer KV sharing specifically as a fine-grained stage-cost control.
Predictive placement and budget selection. Under the single-bottleneck, uniform-saving assumptions, we derive m⋆≈H/Δ, which matches the measured optimum in all five controlled LM-head-surplus experiments. Equal-FLOP placements also yield substantially different iteration times: 257.1 s (Uniform), 251.2 s (Symmetric-Bipolar), and 237.5 s (KV-Pipe).
Efficiency with fewer modifications. At PP=8, converting four layers removes only 1.5% of model FLOPs but reduces iteration time by 9.8%. Compared with fixed 25% sharing, KV-Pipe achieves nearly identical throughput with half as many converted layers.
C3. KV-Pipe's advantage over ECHO-style fixed 25% sharing is not higher peak throughput, but achieving comparable performance with fewer architectural changes through cost-aware budget selection. In LLaMA2-7B, both approaches achieve nearly identical iteration times at PP=8 (−9.8% vs. ~−10.0%), while fixed-ratio sharing is slightly faster at PP=4 (−7.0% vs. −6.3%). At PP=2, the configurations coincide. We will clarify these results in §4.4.
The key advantages of KV-Pipe are: Fewer conversions for comparable speed. At PP=8, four shared layers achieve 237.5 s, versus ~237.0 s with eight layers. Additional conversions provide negligible runtime benefit, so KV-Pipe achieves essentially the same speed with half as many architectural changes. Better quality preservation. LLaMA2-7B quality differences are within observed run variability, so we do not claim a significant quality improvement there. However, the three-run LLaMA3-8B study shows that four-layer sharing maintains PPL close to full attention (6.23±0.02 vs. 6.21±0.02), whereas fixed 25% sharing increases it to 6.28±0.03. This supports limiting unnecessary conversions, although the four-layer budget was not independently optimized for LLaMA3-8B. Adaptation to pipeline imbalance. A fixed 25% ratio ignores stage costs and may over- or under-correct the bottleneck. In our controlled LM-head-surplus experiments, the optimal budget varies from two to six layers, and FIR predicts the measured optimum in all five settings. A single fixed sharing ratio cannot adapt to these changes.