acceptodds
Under review as a conference paper at ICLR 2027

LZIP-P2P: LOSSLESS COMPRESSION SUPERCHARGED GPU COMMUNICATION

Abstract

The rapid growth of large language models (LLMs) has made GPU communication a critical bottleneck. While prior work reduces communication volume via quantization or lossy compression, these approaches introduce numerical errors that can degrade convergence, accuracy, and stability. We present LZIP-P2P, a design that integrates lossless compression directly into GPU point-to-point (P2P) communication primitives without compromising numerical correctness. LZIP-P2P employs a split-send pipeline that exposes transmissible data early and overlaps compression with communication, while preserving high GPU efficiency by operating on large data blocks. In real workloads, LZIP-P2P accelerates RL weight synchronization by up to 47.5% without application changes.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
machine learning systems, GPU communication, lossless compression, distributed LLM training

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_LYmNOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This work proposes LZIP-P2P, a lossless compression framework for GPU point-to-point (P2P) communication to reduce communication overhead without introducing numerical errors. The proposed method introduces a split-send pipeline that overlaps compression and communication. Unlike conventional pipelining methods that divide tensors into smaller chunks, LZIP-P2P exploits the internal stages of floating-point compression. Specifically, it transmits the uncompressed portion of a tensor immediately after the splitting stage while the remaining portion undergoes compression. This allows the method to overlap communication with compression while maintaining GPU efficiency.

The proposed method is integrated into UCCL-P2P without requiring application changes. It also supports different floating-point formats and GPU platforms, including NVIDIA and AMD GPUs. The experimental evaluation covers different tensor sizes, floating-point formats, and resource-constrained settings. The proposed method achieves up to 52.9% higher P2P communication throughput and 47.5% higher weight synchronization throughput for LLM reinforcement learning workloads compared to the baseline.

Critical strengths
  • The proposed method preserves numerical correctness while reducing GPU communication overhead. This is beneficial for distributed training workloads where lossy compression may introduce numerical errors.
  • The evaluation covers multiple GPU architectures, floating-point formats, and communication settings. The implementation also requires no application changes, which improves its practicality.
Critical weaknesses
  • The technical novelty of the proposed method seems limited. The main contribution is a relatively simple split-send pipeline. In addition, the paper does not provide experimental comparisons with recent lossless compression methods, such as ZipNN, NeuZip, DFloat11, and ZipCCL, under comparable settings. This makes it difficult to assess the advantages of the proposed design over alternative lossless compression approaches.
  • The paper does not demonstrate improvements in overall RL training performance. Section 5.3 only evaluates the communication throughput of individual weight tensors, so it is unclear how much the reported throughput improvements contribute to reducing total training time.
Questions for authors
  • Can the authors provide experimental comparisons with recent lossless compression methods, such as ZipNN, NeuZip, DFloat11, and ZipCCL, under comparable settings where feasible, to better demonstrate the effectiveness of the proposed method?
  • Can the authors evaluate the end-to-end performance improvement of RL training, rather than only measuring the communication throughput of individual weight tensors?
4 comments
AuthorsOpenReview ↗

Response to Reviewer

We thank the reviewer for the careful reading and the two concrete requests. Both new experiments are now in the revised PDF. Due to the OpenReview length limit, this response may be split into two posts.

Q1: Comparison with recent lossless compression methods (ZipNN, NeuZip, DFloat11, ZipCCL) under comparable settings.

We added Section 5.3 and Table 3. Like LZip-P2P, all of these methods exploit the low entropy of the BF16 exponent and leave sign and mantissa largely uncompressed. They differ in where and when compression runs, and that is what matters for communication.

Setting: we run each method's released code on one GH200 on the first 32 BF16 weight tensors of at least 16 MB in the released checkpoints of GLM4-9B (32 MiB–1.16 GiB, 4.1 GiB in total) and Qwen3.5-35B-A3B (1 GiB each, 32 GiB in total).

  • ZipNN uses 16 threads, where its own evaluation finds compression throughput to peak.
  • DFloat11's compressor is single-threaded by design and parallelizes only across tensors, so we run it with 16 processes, one tensor each.
  • NeuZip runs on the GPU in both directions.
  • ZipCCL is concurrent work developed independently of ours, and its implementation has not been released, so we could not measure it. It targets collectives on top of NCCL rather than P2P transfers and uses a fixed-length exponent code without overlapping compression with the transfer; its codec could therefore serve as the second stage of our pipeline.
MethodComp. / Decomp.Ratio (GLM / Qwen)Comp. GB/sDecomp. GB/sTransfer, GLM / Qwen (s)
ZipNNCPU / CPU0.663 / 0.6591.5–1.85.3–8.63.95 / 23.9
NeuZipGPU / GPU0.674 / 0.673245–35456–730.139 / 1.17
DFloat11CPU / GPU0.676 / 0.6760.007486–526180 / 320
LZip-P2PGPU / GPU0.677 / 0.676663–775785–9820.072 / 0.544
Uncompressed–1.00––0.089 / 0.687

("Transfer" = compression + compressed bytes over one 400 Gb/s NIC + decompression)

Findings:

  1. The compression ratios are close (0.659–0.677). All methods approach the same exponent entropy, so the ratio does not separate them.
  2. LZip-P2P fits P2P communication best. What separates them is whether both compression and decompression keep up with the network (50 GB/s at 400 Gb/s), since in RL weight synchronization the weights change every step and must be re-encoded each time. Each baseline falls short for a reason rooted in the setting it was designed for.
    • ZipNN targets model storage and download, so it compresses and decompresses on the CPU, far more slowly than a GPU codec;
    • DFloat11 targets inference, where a model is compressed once offline; its compressor therefore runs on a single CPU thread, and only decompression runs on the GPU, one transformer block at a time in every forward pass.
    • NeuZip targets memory-efficient training and inference with layer-by-layer decompression on the GPU. However, decoding a tensor takes about as long as transmitting it uncompressed, which cancels the 33% byte reduction: NeuZip takes 0.139 s for GLM4-9B, slower than sending uncompressed (0.089 s).
    • LZip-P2P's DietGPU-based codec instead runs both directions on the GPU at 663–982 GB/s, more than 13× the link rate, by fusing the split with the exponent histogram and coding exponents in independent 4 KiB ANS blocks. It is therefore well suited to P2P transfer.
  3. Our contribution is orthogonal to the codec. The split-send pipeline is the communication-side design, and any of these exponent coders could serve as its second stage.

On novelty: we agree the split-send pipeline is simple, and we see that as a strength. The key step is aligning the pipeline with the compressor's internal stages instead of with data size. That alignment is what makes lossless compression profitable for P2P transfers, and Table 3 shows that a codec alone does not provide this benefit.

Q2 (continued): End-to-end RL training performance, not only per-tensor throughput.

We added Section 5.5 and Table 4. We integrated LZip-P2P into the open-source miles RL framework as a P2P weight-transfer backend:

We run GRPO on Qwen3-4B on two nodes with one GH200 each: the Megatron trainer on one node, a single SGLang engine on the other, so every update crosses the network. Each step generates 2 prompts × 4 samples with responses of at most 32 tokens and trains on a global batch of 8, in synchronous GRPO and in asynchronous RL. The only difference between runs is the weight-transfer backend: miles' default Mooncake or LZip-P2P:

SettingBackendWeight update (s)Rollout (s)Train (s)Step (s)
Qwen3-4B, syncMooncake (miles default)0.2200.2430.6122.134 ± 0.049
LZip-P2P0.173 (−21.4%)0.2440.6001.979 ± 0.054 (−7.3%)
Qwen3-4B, asyncMooncake0.2170.248*0.4881.087 ± 0.042
LZip-P2P0.178 (−18.0%)0.237*0.4640.955 ± 0.024 (−12.1%)

The weight update includes format conversion and the completion fence. *In asynchronous runs, rollout overlaps with training and is off the critical path.

Findings:

  1. Weight updates are 18–21% faster than with Mooncake. The gain is smaller than in the per-tensor benchmark because the update also converts weights to the engine's layout, sends small tensors uncompressed, and waits for each 1 GB bucket to be decompressed before converting the next.
  2. Step time. The end-to-end effect scales with the share of the step spent in weight updates. We use small batches, as in asynchronous RL that updates the weights after every small batch to keep rollouts nearly on-policy. With LZip-P2P, step time drops by 7.3% in synchronous GRPO and 12.1% in asynchronous RL.
  3. No effect on the training trajectory. The transfer is lossless, and miles' per-step weight-equality check confirms the engine receives bit-identical weights.

We have narrowed the application-level claims in the paper accordingly.

Thank you for the additional experiments.

In Table 4, the reductions in total RL step time (155 ms and 132 ms) are substantially larger than the reductions in weight-update time (47 ms and 39 ms). Could the authors clarify which additional components account for these differences?

AuthorsOpenReview ↗

Clarify additional components account

Thank you for checking these numbers. The step-time reductions in Table 4 (155 / 132 ms) include the effect of replacing the transport backend, not only of compression. To separate the two, the table below adds a run with the same UCCL P2P transport and compression disabled:

SettingBackendWeight update (s)Rollout (s)Train (s)Step (s)
Qwen3-4B, syncMooncake0.2200.2430.6122.134
UCCL P2P (no compression)0.2080.2440.6122.030
LZip-P2P0.1730.2440.6001.979
Qwen3-4B, asyncMooncake0.2170.248*0.4881.087
UCCL P2P (no compression)0.2070.233*0.4561.035
LZip-P2P0.1780.237*0.4640.955

* In asynchronous runs, rollout overlaps with training and is off the critical path.

  • Replacing Mooncake with UCCL P2P (no compression) shortens the step by 104 ms (sync) and 52 ms (async). Besides the three columns above, a step also includes the log-probability pass, releasing training memory before the update, and framework work around the update. Replacing the Mooncake backend with UCCL P2P reduces these additional overheads. A likely reason: in miles, the Mooncake backend keeps the trainer's copy of the model in host memory, copies it from the GPU at every update, and runs its own transfer engine next to the trainer process. UCCL P2P keeps the copy in GPU memory and writes it directly with GPUDirect RDMA, which leaves less host-side work to slow down the other phases.
  • Adding compression (UCCL P2P → LZip-P2P) shortens the update by 35 ms (sync) and 29 ms (async).

Why the end-to-end effect is modest here. Our added experiments train on a single GH200 (96 GB). Full-parameter training with Adam needs about 18 bytes per parameter, so Qwen3-4B is the largest model that fits; each update transfers 8.0 GB of weights. At this scale the weight update is only about 10% (sync) to 20% (async) of the step, so an 18–21% faster update gives a small step-time saving.

In large-scale RL, the update takes a much larger share of the step. Every step must move the full model from the trainers to all inference engines, and this takes seconds to minutes:

  • SGLang reports 53 s with NCCL broadcast for Kimi-K2 (1T parameters) and 10.8 s for Qwen3-235B-A22B, noting that "during weight transfer, the entire RL training comes to a halt" [1].
  • Prime Intellect reports that syncing weights after every step "stayed stuck at 60–90 seconds" while their step times were under five minutes, which puts the update at roughly 20–30% of the step [2].

Asynchronous RL, which updates the weights after every small batch to stay nearly on-policy, makes updates even more frequent. In these settings the update-time saving of LZip-P2P translates into a larger end-to-end saving.

[1] https://lmsys.org/blog/2026-04-29-p2p-update/

[2] https://primeintellect.ai/blog/nixl-modelexpress-weight-transfer

Reviewer_gDWiOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

LZIP-P2P integrates lossless floating-point compression into GPU point-to-point communication. The main contribution from the paper is the 'split-send pipeline'. After separating the sign and fraction bits from the exponent bits, the sign and fraction bits are immediately communicated while the exponent is compressed. This allows for overlapping between compression and communication, while maintaining large compression blocks which can efficiently utilize GPU parallelism. The implementation supports five floating-point formats, and is compatible with both NVIDIA and AMD GPUs. Evaluation of two-node platforms (NVIDIA/EFA and AMD/RoCEv2) reports up to 52.9% higher throughput for bfloat16 tensors of sizes 8MB-1GB of data uniformly distributed between [-1,1]. Additional experiments evaluate throughput increase compared to UCCL-P2P for two representative models, as well as performance through memory and SM availability constraints, and different data formats.

Critical strengths
  1. The design exploits a clear and practical scheduling insight, that being the sending of uncompressed fraction and sign bits while compressing the exponent bits, leading to an overlap between communication and compression. The comparison with chunk-based pipelining and encode-send methods support the value of the 'split-send' scheduling model proposed.
  2. The support between many data types and floating point formats show substantial implementation and portability work. This wide support strengthens the practical contribution.
Critical weaknesses
  1. The evaluation focuses almost entirely on communication throughput, and the ablation study does not make it clear how receiver-side decompression and placement into the destination buffer are affected through their proposed scheduling model.
Questions for authors
  1. What portions of communication are throughput timings derived from? Is receiver decompression and placement into the destination buffer included in all throughput timings? If they are not included, I would like to see how these affect the communication throughput compared to encode-send and chunk-based pipelining models.
  2. How is communication completeness handled? How does the receiver ensure that both payload portions have arrived prior to reconstruction?
1 comment
AuthorsOpenReview ↗

Response to Reviewer

We thank the reviewer for the positive assessment and the precise questions about receiver-side behavior.

Q1: What portions of communication are the throughput timings derived from? Are receiver decompression and placement into the destination buffer included?

Yes. Every reported throughput number covers compression, transfer and receiver-side decompression into the destination buffer, for LZip-P2P, encode-send and the naive pipeline alike.

Q2: How is communication completeness handled? How does the receiver ensure both payload portions have arrived before reconstruction?

The two portions travel as separate RDMA operations of the same message, which the sender tracks under one message id. Only after all completions of both portions have arrived does the sender send a small per-message metadata record to the receiver; under the reliable RDMA transport, a completion means the data has been placed at the receiver. The receiver starts decompression only when it sees this metadata record, so both portions are already in its staging buffer. The kernel writes directly into the destination buffer, and recv completes when the kernel finishes. The receiver therefore never reconstructs from a partial payload. We added this description to Section 4.

Reviewer_oNLhOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper’s main contribution is a stage-aligned pipeline that overlaps lossless compression with GPU point-to-point communication. Its large-tensor gains are substantial: 72.2 versus 47.2 GB/s for a synthetic 1 GB transfer, and up to 47.5% higher throughput for a selected 214 MB model-weight tensor. These are not minor improvements. However, the experiments primarily measure individual transfers, not the effect on a complete training workload. The benefit also falls sharply for smaller tensors. The systems idea is promising, but its application-level significance and the advantage of this particular pipeline design need stronger evidence.

Critical strengths
  1. Substantial large-tensor gains without numerical loss. The reported 1 GB transfer improves from 47.2 to 72.2 GB/s, and a selected 214 MB weight tensor gains up to 47.5%. These are potentially consequential systems improvements, not marginal score changes.
  2. Evaluation across two hardware stacks. Results on both NVIDIA- and AMD-based systems provide initial evidence that the pipeline is not tied to one device configuration.
Critical weaknesses
  1. Clarify the novelty. The paper builds on an existing lossless exponent-compression approach. Its distinct contribution appears to be when the two streams are transmitted and how compression overlaps with communication. Please make that boundary explicit and provide stage-level timing that shows why this schedule improves on a well-optimized conventional pipeline.
  2. Measure the complete application. Figure 5 reports selected weight-tensor transfers, not whole-model synchronization or reinforcement-learning iteration time. Please report at least one complete workload result, including compression, transfer, decompression, and any interference with GPU computation. Otherwise, the application-level claim should be narrowed.
  3. Characterize where the method helps. The 1 GB gain is compelling, but the reported improvement is about 8% at 16 MB and becomes a 6% regression at 8 MB. Please show the tensor-size distribution of the target workloads and an aggregate result across all transfers. A measured size-dependent bypass rule may be preferable to compressing every eligible message.
  4. Strengthen the pipeline ablation. The comparison with encode-then-send and one fixed 8 MB chunked alternative does not fully isolate the proposed scheduling choice. A chunk-size sweep and a breakdown of compression, waiting, transmission, and decoding time would establish whether the stage-aligned boundary—rather than the particular alternative implementation—causes the gain.
  5. Report measurement reliability. The paper should specify repetitions, warm-up, synchronization and timing boundaries, and variation across runs. This is particularly important for the roughly 8–10% gains, whose practical significance cannot be judged from point estimates alone.
  6. Test realistic resource contention. Restricting available GPU resources is useful, but it does not replace measuring concurrent training computation and multiple communication flows. Please report the full sender/receiver resource cost and show whether the speedup persists when communication competes with the computation present in the intended workload.
Questions for authors
  1. Were complete model-synchronization or training-iteration timings already measured? If so, what is the aggregate result across all tensor sizes? If not, this requires a new experiment rather than a rebuttal explanation.
  2. What are the repetition counts, timing boundaries, and run-to-run variation? Do reported transfer times include receiver-side decompression?
1 comment
AuthorsOpenReview ↗

Response to Reviewer

We thank the reviewer for the detailed and constructive review. We agree that application-level evidence was missing, and we added new experiments and updated the paper PDF accordingly.

Q1: Were complete model-synchronization or training-iteration timings measured? What is the aggregate result across all tensor sizes?

We added training-iteration experiments: GRPO with the miles RL framework, with the Megatron trainer and the SGLang engine on separate nodes, where every weight update transfers the whole model, including tensors of all sizes: 398 tensors totaling 8.0 GB for Qwen3-4B and 399 tensors totaling 16.4 GB for Qwen3-8B. The experiments run on two nodes, each with one NVIDIA GH200 and one 400 Gb/s InfiniBand NIC.

  • Size distribution. Tensors ≥16 MB are 45% of the tensors but carry 95% (Qwen3-4B) and 96% (Qwen3-8B) of the bytes. Tensors <1 MB are 36% of the count but <0.01% of the bytes.
  • Aggregate result. LZip-P2P shortens the whole-model weight update by 18–21% relative to miles' default Mooncake backend:
ModelSettingMooncake (s)LZip-P2P (s)Reduction
Qwen3-4Bsynchronous GRPO0.2200.173−21%
Qwen3-4Basynchronous RL0.2170.178−18%
Qwen3-8Basynchronous RL0.4090.323−21%

This is lower than the per-tensor gains because the update also converts weights to the rollout engine's layout, sends small tensors uncompressed, and waits for each 1 GB bucket to be decompressed before converting the next. Details are in Section 5.5 of the revised PDF.

Q2: What are the repetition counts, timing boundaries and run-to-run variation? Do reported transfer times include receiver-side decompression?

  • Timing boundaries. Every reported throughput number covers compression, transfer and receiver-side decompression into the destination buffer, for LZip-P2P, encode-send and the naive pipeline alike.
  • Repetitions. Each microbenchmark point is the mean of 5 independent runs, each timing 128 back-to-back transfers after 1 warm-up transfer.
  • Variation. On the GH200 platform, the run-to-run coefficient of variation is ≤1.5% for every compressed size and ≤0.15% from 128 MB up. The gains at all sizes are far outside the noise.

We added these details to Section 5.1.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.