- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
LCT is a lossless BF16 tensor-compression framework for storing weights, activations, and optimizer states during training. It combines bounded Huffman coding of exponents with GPU kernels, fused pointwise updates, and a shared overflow buffer for repeated encoding and decoding.
Critical strengths
- Fast repeated encoding addresses a practical limitation of decode-oriented compression methods.
- Fused operators and GPU-side overflow allocation make the approach useful beyond a standalone codec.
- Codec benchmarks span several GPU architectures, with end-to-end memory savings demonstrated in two models.
Critical weaknesses
- End-to-end evaluation uses BF16 Muon setups; standard AdamW training with FP32 optimizer states is not evaluated.
- Qwen measurements cover only five steps after warm-up, leaving sustained overhead and overflow-buffer behavior unclear.
- The authors disclose agent-driven kernel optimization, but kernel-level ablations are missing to clarify which design choices support the “GPU-friendly” claim.
- The presentation feels overly AI-polished and generic. Figure 2, in particular, adds little technical insight beyond the surrounding prose.
Questions for authors
Do memory savings and runtime overhead remain stable throughout a full Qwen fine-tuning run?
Thank you for taking your time to review our paper and your thoughtful comments.
Loss over a full Qwen fine-tuning run
We fine tune Qwen3-4B-Base for 1000 steps with and without LCT. We use a GH200 system and increase the sequence length to 16384 for a more realistic fine-tuning setup.
The final loss difference is 0.000358. This difference is within normal kernel nondeterminism and multiprocess dataloader noise. We also track the memory usage throughout fine-tuning in new Figure 7, which remains consistent throughout the run and drops 35.2% from 77.67GiB to 50.31GiB. The time taken per step increases from 2.953s/step to 3.174s/step, a 7.5% increase. The compression ratio of LCT is very consistent, the peak overflow under 140MiB throughout the 1000 steps.
Where the kernel speed comes from
We conduct an ablation study to show how our design choices affected the final performance. On 1 GiB of BF16 Gaussian data we remove one mechanism at a time relative to the final kernels to show their impact.
The three largest factors are using a stream-major payload instead of a coalesced, and splitting exponent extraction from sign-mantissa packing costs 68% as well as our subsampled center estimation. These reduce the memory traffic which is the largest bottleneck. Encoding / decoding multiple symbols at a time reduces the number of fetches requested leading to better overlap. Our swizzle method keeps words contiguous in memory for better coalescing while breaking apart correlations in the input tensor. Finally, prefetching the decode output allows the next symbol to be decoded while the previous is being written back to memory.
Fused updates and shared overflow
Compressing state only pays if the update path stays cheap, so we compared decode → scaled-add → encode against an update performed inside the codec. Fusion is 1.46× faster on 1 GiB of Gaussian data and 1.24× on a real Qwen FFN matrix, but 0.94× on a small Q-projection, so the gain follows tensor size. In native NanoGPT training, the shared overflow buffer removes host synchronization and cuts step time from 8.836 s to 8.703 s, for 57.1 MiB of reserved peak memory. A separate sweep over activation granularity favors a size-dependent policy, which we will report in the revision.
Adam optimizer and FP32 results
Our response to Reviewer 5LRk measures both Adam moments across OLMo training. Since FP32 uses the same exponent setup as BF16, our method can easily be applied to FP32 tensors. On a FP32 tensor, we find the same compression ratio on the exponent is the same as BF16, but overall compression ratio is lower as the mantissa is larger. Below we show the time to encode / decode a 250M element gaussian tensor: