acceptodds
Under review as a conference paper at ICLR 2027

LCT: Losslessly Compressed Tensors with GPU Friendly Execution for Training

Abstract

Lossless tensor compression could reduce memory usage during training, but training requires tensors to be repeatedly re-encoded. We introduce Losslessly Compressed Tensors (LCT), a GPU execution framework designed for this repeated encode–decode regime. LCT exploits the natural redundancy in floating point data to reduce storage for weights, activations and optimizer states. Highly optimized GPU kernels implement Huffman coding with precomputed codebooks for low compression and decompression overhead, essential for training. Our analysis characterizes LCT's storage costs and adaptation to magnitude shifts. LCT achieves a compression ratio of up to , and is demonstrated in training and finetuning settings. On a 162M-parameter NanoGPT, LCT reduces memory usage by for a step-time increase. In Qwen3-4B fine-tuning, LCT saves up to of peak memory, enabling a 3.5X larger context length in training on the same GPU.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Lossless compression, Training models, Triton, Memory reduction

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_94owOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

LCT is a lossless BF16 tensor-compression framework for storing weights, activations, and optimizer states during training. It combines bounded Huffman coding of exponents with GPU kernels, fused pointwise updates, and a shared overflow buffer for repeated encoding and decoding.

Critical strengths
  • Fast repeated encoding addresses a practical limitation of decode-oriented compression methods.
  • Fused operators and GPU-side overflow allocation make the approach useful beyond a standalone codec.
  • Codec benchmarks span several GPU architectures, with end-to-end memory savings demonstrated in two models.
Critical weaknesses
  • End-to-end evaluation uses BF16 Muon setups; standard AdamW training with FP32 optimizer states is not evaluated.
  • Qwen measurements cover only five steps after warm-up, leaving sustained overhead and overflow-buffer behavior unclear.
  • The authors disclose agent-driven kernel optimization, but kernel-level ablations are missing to clarify which design choices support the “GPU-friendly” claim.
  • The presentation feels overly AI-polished and generic. Figure 2, in particular, adds little technical insight beyond the surrounding prose.
Questions for authors

Do memory savings and runtime overhead remain stable throughout a full Qwen fine-tuning run?

1 comment

Thank you for taking your time to review our paper and your thoughtful comments.

Loss over a full Qwen fine-tuning run

We fine tune Qwen3-4B-Base for 1000 steps with and without LCT. We use a GH200 system and increase the sequence length to 16384 for a more realistic fine-tuning setup.

DenseLCT
Initial loss2.3562972.356297
Final loss2.2922642.292622
Time per step (s)2.9533.174
Peak memory (GiB)77.6750.31

The final loss difference is 0.000358. This difference is within normal kernel nondeterminism and multiprocess dataloader noise. We also track the memory usage throughout fine-tuning in new Figure 7, which remains consistent throughout the run and drops 35.2% from 77.67GiB to 50.31GiB. The time taken per step increases from 2.953s/step to 3.174s/step, a 7.5% increase. The compression ratio of LCT is very consistent, the peak overflow under 140MiB throughout the 1000 steps.

Where the kernel speed comes from

We conduct an ablation study to show how our design choices affected the final performance. On 1 GiB of BF16 Gaussian data we remove one mechanism at a time relative to the final kernels to show their impact.

AblationEncode slowdownDecode slowdown
Stream-major rather than coalesced payload104.4%37.6%
Separate exponent/sign-mantissa extraction67.5%—
Full-input rather than sampled centre estimation69.7%—
Single-symbol processing in the respective kernel23.3%8.2%
No lane swizzle3.2%10.6%
No decode prefetch—8.1%

The three largest factors are using a stream-major payload instead of a coalesced, and splitting exponent extraction from sign-mantissa packing costs 68% as well as our subsampled center estimation. These reduce the memory traffic which is the largest bottleneck. Encoding / decoding multiple symbols at a time reduces the number of fetches requested leading to better overlap. Our swizzle method keeps words contiguous in memory for better coalescing while breaking apart correlations in the input tensor. Finally, prefetching the decode output allows the next symbol to be decoded while the previous is being written back to memory.

Fused updates and shared overflow

Compressing state only pays if the update path stays cheap, so we compared decode → scaled-add → encode against an update performed inside the codec. Fusion is 1.46× faster on 1 GiB of Gaussian data and 1.24× on a real Qwen FFN matrix, but 0.94× on a small Q-projection, so the gain follows tensor size. In native NanoGPT training, the shared overflow buffer removes host synchronization and cuts step time from 8.836 s to 8.703 s, for 57.1 MiB of reserved peak memory. A separate sweep over activation granularity favors a size-dependent policy, which we will report in the revision.

Adam optimizer and FP32 results

Our response to Reviewer 5LRk measures both Adam moments across OLMo training. Since FP32 uses the same exponent setup as BF16, our method can easily be applied to FP32 tensors. On a FP32 tensor, we find the same compression ratio on the exponent is the same as BF16, but overall compression ratio is lower as the mantissa is larger. Below we show the time to encode / decode a 250M element gaussian tensor:

FormatEncodeDecodeCompression ratioStorage reduction
BF160.488 ms0.504 ms1.454×31.23%
FP320.755 ms0.782 ms1.185×15.61%
The storage reduction is roughly half for FP32, and the encode / decode time is about 1.5× slower since the size of the tensor is twice as large for FP32 and we are memory limited.
Reviewer_5LRkOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes a GPU execution framework, named LCT, which exploits the natural redundancy in floating-point data to reduce storage for weights, activations, and optimizer states. LCT compresses the exponent bytes of BF16 tensors using bounded Huffman coding with escape codes, while storing sign and mantissa information unchanged. It combines this codec with GPU-optimized Triton kernels, fused pointwise operations for optimizer updates, a shared GPU buffer for variable-sized compressed data, and PyTorch autograd integration. The evaluations through codec benchmarks, pointwise operator benchmarks, NanoGPT training, and full-parameter fine-tuning of Qwen3-4B show the effectiveness of the propose approach.

Critical strengths
  1. It offers an alternative to lossy quantization and CPU offloading by preserving the exact BF16 representation.
  2. The paper analyzes storage overhead under distribution mismatch rather than reporting compression ratios alone.
  3. The evaluation covers individual kernels and two training settings.
Critical weaknesses
  1. The training-level evaluation focuses on NanoGPT and Qwen3-4B, while much of the codec analysis uses Gaussian synthetic tensors or selected tensor groups from Nemotron-H-8B. They can not fully establish how consistently LCT performs across diverse architectures.
  2. The empirical evaluation could better characterize how mismatched codebooks affect actual memory savings and execution time during training.
  3. A more explicit connection between the analytical results and the observed end-to-end performance would strengthen the paper.
Questions for authors

How should users choose a coding distribution and storage budget for a new model? Can the authors provide a practical selection rule or demonstrate robustness when the codebook is mismatched to the observed exponent distribution?

2 comments

Thank you for taking the time to review our paper and your thoughtful comments.

Compression across training and Adam moments

To study the behavior of LCT in full training, we evaluate LCT on checkpoints from the OLMO 3 7B model. This is a open source model that has public checkpoints from initialization to end of training. We pick 13 OLMo 3 7B checkpoints evenly spaced covering pre-training, long-context adaptation and fine-tuning to study the evolution of LCT. OLMO 3 uses the ADAM optimizer rather than Muon. For the second moment, we found our standard codebook used for momentum to perform suboptimally, in particular, the tail is much heavier. We calibrate a new codebook on a checkpoint 1% through training and use it for the second moment buffer, improving compression from about 1.1X to 1.3X. Retained state falls from 40.784 GiB to 27.5–28.0 GiB, a 31.4–32.5% reduction.

Final-checkpoint componentCompression ratio
Weights1.463×
Adam first moment1.436×
Adam second moment1.308×
Weights + both moments1.462×

We track the total size of the compressed weights and momentum buffers:

Training stageStepCompressed memory / GiB
Pretraining14,00027.521
Pretraining71,00027.710
Pretraining141,00027.820
Pretraining353,00027.911
Pretraining707,00027.926
Pretraining1,060,00027.893
Pretraining1,413,81427.885
Midtraining1,00027.951
Midtraining24,00027.930
Midtraining47,68427.975
Long-context1,00027.931
Long-context6,00027.878
Long-context11,92127.893

The saving is stable across the trajectory rather than specific to final weights, with a slight increase towards the end of training. OLMo stores these tensors in FP32 natively, so these are BF16 projections.

These experiments are run on a GH200 system, compared to our other results which mainly use a RTX 3090.

Choosing a distribution and budget

For all of the experiments in the paper, we the Laplace distribution with low noise, which models typical tensors fairly well. If the tensor has exact nonzeros (e.g. ReLU activations), we manually specify this in the distribution setup, recall LCT's codebook allows for exact zeros.

When compressing OLMO's second momentum buffer, we found that the Laplace distribution was not a good fit for the tail, so we calibrated a new codebook using the empirical distribution on a checkpoint 1% through training, and used for the rest of the checkpoints. If the Muon optimizer is used, there is no second momentum buffer so this is not needed.

If a poor distribution is chosen, the compression ratio will be worse due to more escape codes and the overflow buffer.

Thank you for the responses. I have read the rebuttal and have no further questions.

Reviewer_eb7mOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes LCT, a lossless BF16 tensor codec targeted at training, where tensors must be repeatedly re-encoded. It compresses only the 8-bit exponent with length-limited Huffman codes plus an escape mechanism, keeps sign/mantissa bytes raw, packs symbols into fixed-budget streams with raw overflow tails, and shifts a precomputed codebook by a sampled exponent center to track magnitude drift. The system includes Triton encode/decode kernels, fused pointwise operators for optimizer updates on compressed state, a host-synchronization-free shared overflow buffer, and PyTorch autograd integration. Validation covers codec throughput against SplitZip, DFloat11, ZipNN and nvCOMP (LCT encodes fastest, ~1.45× ratio), fused-operator timings, a 162M NanoGPT run (24.6% peak-memory reduction, 7.2% step-time increase, matching validation loss), and Qwen3-4B full fine-tuning on a 24 GB GPU (22–24% memory reduction, enabling sequence lengths up to 3500 where the baseline OOMs above 1000).

Critical strengths

S1. Practical problem, correct framing. GPU memory is a major bottleneck in LLM training, and LCT is lossless, so it adds no numerical error. The paper correctly identifies that existing lossless codecs (DFloat11, ZipNN) are decode-centric and CPU-encoded, making them unusable for training where state is rewritten every step. This encode-bandwidth framing is the main conceptual contribution.

S2. Strong GPU-oriented engineering. Beyond the codec itself, parallel stream encoding, fused pointwise operators (halving memory traffic for optimizer updates), and the host-synchronization-free OverflowBuffer address the practical overheads that would otherwise block training integration.

S3. Convincing codec-level performance. Table 4 shows 2.454 ms encode time for 1 GiB versus 5.850 ms for SplitZip, with a higher compression ratio (1.455× vs 1.333×), and orders of magnitude over DFloat11 and ZipNN. The advantage holds across V100, A40, and H100.

S4. End-to-end results with sound methodology. NanoGPT and Qwen3-4B experiments show measurable peak-memory reductions at modest step-time cost. Baselines are memory-optimized before comparison, and losslessness is verified on NanoGPT via matching validation loss.

Critical weaknesses

W1. No comparison against the methods a practitioner would actually choose . The introduction positions LCT against activation checkpointing, low-bit activation compression (ActNN), and 8-bit optimizer states, but none appears in any experiment. LCT delivers at most ~1.44× on compressed tensors and ~1.33× in end-to-end peak memory for a 7–8% slowdown. Full activation recomputation gives far larger savings at comparable compute cost, and 8-bit optimizers give 2× on state with negligible quality loss in practice. Without a direct comparison on the same setup, the central claim that lossless compression is a "practical training-memory primitive" is unsupported.

W2. Modest gains, narrow evidence. All training results are on a single RTX 3090. The headline "3.5× longer context" is really "the baseline OOMs on a 24 GB card above 1000 tokens." The Qwen3-4B experiment reports no loss or accuracy, so losslessness is asserted rather than shown for the larger model. Weight and momentum compression contribute <1% of the NanoGPT saving (138 MiB), and no breakdown is given for Qwen.

W3. Robustness to distribution choice is untested. The method depends on a user-supplied distribution and adapts only to scale, not shape. Robustness to a mis-specified model is evaluated only on synthetic sources (Table 2). Adam is listed as implemented but never evaluated, even though second-moment buffers have very different exponent statistics from the cases studied.

Questions for authors

Clarification requests (answers could affect my assessment):

  1. On the NanoGPT setup, what are peak memory and step time for (a) full activation checkpointing and (b) 8-bit optimizer states, measured identically to Table 6?
  2. For Qwen3-4B, what is the final loss with and without LCT, and what fraction of the ~4 GB saving comes from activations vs. weights vs. momentum?
  3. What compression ratios and overflow rates does LCT achieve on Adam first and second moments?
  4. In Table 4, SplitZip's calibration is excluded while its host-synchronizing allocation is included. How much of the 2.38× encode gap remains if SplitZip is given a preallocated buffer?
  5. Since GEMMs reconstruct dense weights, is the weight-compression memory benefit only transient between uses?

Suggestions (not required for assessment):

  • Multi-GPU / FSDP results.
  • An automatic distribution selector instead of a user-specified model.
  • Evaluation of the Medium/High budgets under distribution shift.
1 comment

Thanks for the positive assessment of the encode-bandwidth framing, the GPU implementation and the experimental methodology.

Comparison with activation checkpointing

LCT is compared with activation checkpointing which is a common lossless memory saving technique. We ran four configurations on the RTX 3090: dense, LCT, activation checkpointing, and both together. Checkpointing is applied to the attention and FFN layers. In the combined configuration we compress the checkpoint inputs as well.

ModelConfigurationPeak allocated (MiB)Memory vs denseStep time (s)Step time vs dense
NanoGPTDense16,058.4—7.146—
NanoGPTLCT11,966.9−25.5%7.693+7.7%
NanoGPTCheckpointing5,228.6−67.4%9.683+35.5%
NanoGPTCheckpointing + LCT4,593.3−71.4%9.815+37.3%
Qwen3-4BDense21,327.8—3.696—
Qwen3-4BLCT16,246.1−23.8%3.843+4.0%
Qwen3-4BCheckpointing18,223.8−14.5%3.815+3.2%
Qwen3-4BCheckpointing + LCT14,548.8−31.8%3.906+5.7%

The memory saved per unit of step time is better for LCT than checkpointing, and LCT further can be combined with checkpointing to even lower memory usage.

Qwen savings

We forgot to include this in the main paper. Here, we have the memory savings from LCT on Qwen3-4B broken down by component:

Compressed componentsTime / sPeak memory / MiBEffective compression ratio
None3.69621327.81.000×
Activations3.75219807.11.077×
+ Weights3.80118253.51.168×
+ Momentum buffers3.84316244.21.313×

A new larger finetuning run using long sequence lengths on a GH200 for 1,000 steps are in our response to Reviewer 94ow, including final loss which is nearly identical between the LCT and uncompressed runs.

Memory from reconstructed weights

LCT reconstructs full weights on demand. However, this memory is transient and released immediately after the laye is completed so the memory benefit is not lost. Only 1 block's weight, activation and momentum buffers are decompressed at a time, the rest of the tensors stay compressed. For a transformer with many layers, the decompression only adds a few hundred MiB of memory above a completely compressed implementation.

SplitZip with preallocated storage

We test SplitZip with the same preallocated buffer as in LCT. We modify their kernels to support our buffer system. On the 1 GiB Gaussian benchmark, giving SplitZip a preallocated buffer speeds encoding from 5.861 ms to 5.837 ms. LCT with its shared buffer takes 2.394 ms. SplitZip benefits from the preallocated buffer, but is still slower than LCT.

Adam moments

We investigate the performance of LCT on OLMo, including Adam buffers in our response to Reviewer 5LRk.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.