acceptodds
Under review as a conference paper at ICLR 2027

Compact Activation Storage for ReLU Feed-Forward Networks

Abstract

Reducing memory use in large language model (LLM) training is an ongoing challenge. Activation checkpointing saves memory by recomputation, while lossy compression changes the stored activations. The reemergence of sparse and activations creates a third opportunity: losslessly compressing the activation cache itself. We present BitSparse, a GPU-resident lossless compression format for sparse activations. Compression and reconstruction run in custom Triton kernels fused into an autograd function, shrinking the activation cache while recovering it exactly for the backward pass. Bfloat16 (BF16) and FP8 activations are supported. Optional sign-bit packing further increases compression up to close to the theoretical limit. We analyze its compression ratio and compare it to the theoretical optimum. BitSparse lowers per-token VRAM by 31% and 30% Modded-NanoGPT and Nemotron-H respectively, with overheads of no more than 1% at practical sequence lengths. Additionally, a new preallocated buffer system avoids host synchronization during training when variable size tensors are allocated.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Memory Reduction, Compression, LLM training

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_bYSoOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes BitSparse, a lossless activation compression method for LLM training with ReLU-family activations. It stores nonzero values together with masks and offsets, and implements efficient Triton kernels for compression and reconstruction. The method is evaluated on synthetic MLPs and two language models, with comparisons against several compression and checkpointing baselines.

Critical strengths
  1. Reducing activation memory is an important problem for efficient LLM training.
  2. BitSparse achieves substantial memory savings with little runtime overhead in the reported benchmarks.
  3. The source code is provided as supplementary material, supporting reproducibility.
Critical weaknesses
  1. The method is limited to ReLU-family activations with exact zeros, restricting its applicability to LLMs using other activations such as SwiGLU.
  2. The evaluation relies heavily on synthetic MLP benchmarks. End-to-end training is demonstrated only on a 162M model, while FP8 LLM benchmarks and end-to-end results for larger models (e.g., 8B) are missing.
  3. There are formatting and typographical issues. The "BitSparse:" prefix in the PDF title is missing from the OpenReview submission title. Table captions lack the required spacing below them. Line 320: "BisSparse" should be "BitSparse"; line 255: "there no" should be "there are no".
Questions for authors
  1. How is memory bandwidth utilization calculated in Table 2? The reported values at 0.9 sparsity seem to correspond to 0.8 sparsity instead.
  2. Why does the dense baseline's memory usage differ by approximately 1 GiB between 0.5 and 0.9 sparsity in Tables 3 and 8, whereas Tables 4 and 9-11 report identical values across sparsity levels?
  3. How is the preallocated TensorBuffer capacity determined? What happens if the number of nonzero activations exceeds its capacity during training?
2 comments

Thank you for taking the time to review our paper and check for issues and improvements.

Nemotron-H-8B FP8 fine-tuning

We compare Nemotron-H-8B full finetuning runs on a GH200 GPU for 2,000 updates / 32.8M tokens with sequence length 8,192 and batch size 2. The results in BF16 are given in our response to reviewer 43ev. Here, we give analagous results using FP8 activations and parameters with BF16 optimizer state. The GH200 supports FP8 better than our previously used RTX 3090.

MethodPeak memoryTime/updateFinal validation loss
Dense FP850.004 GiB3.8288 s2.263537
BitSparse FP843.983 GiB3.8560 s2.263597
BitSparse-SB FP843.880 GiB3.8652 s2.263515

Memory usage is reduced by 12.0–12.2% with a slowdown of 0.71–0.95%. The final validation loss is essentially unchanged up to 4 significant figures. Combined with the BF16 full finetuning results, this demonstrates that BitSparse is effective for real BF16 and FP8 fine-tuning.

Formatting corrections

Thank you for finding the formatting issues. We have corrected the title, table captions, and typos in the final version of the paper.

In Table 2, the bandwidth calculation did indeed use a sparsity of 0.8. Thank you for catching this issue.

In Table 3 and 8, the values at sparsity 0.9 are the correct values (11344 and 10039). This was a data entry error, and we will correct this in the final version of the paper. The percentage changes in the main text already reflected the correct values.

Size of the Tensor Buffer.

As we note in the paper, the sparsity is very consistent during training. Theoretically, the TensorBuffer can be directly sized with a small amount of overhead for the worst case sparsity by adding a few percent headroom to its size. A simpler approach is to make the TensorBuffer take up most of the spare memory after accounting for all other memory used by the model (parameters, gradients, optimiszer state etc) since they are always fixed. This gives close to the same size as if no TensorBuffer was used.

If the TensorBuffer runs out, BitSparse will stop with an error instead of causing numerical issues or truncating values.

Thank you for the reply. Weakness 1 is unaddressed, and I think it is a fundamental weakness of this paper that cannot be resolved with a rebuttal. Reviewer 43ev also has this concern. I maintain my rating.

Reviewer_xCAWOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes BitSparse, a GPU-resident lossless format for sparse activations. The method optimizes training by reducing the memory usage of activations, enabling a larger micro-batch size. Consequently, fewer steps are needed for a given dataset, improving end-to-end performance. The paper claims that per-token VRAM usage is lowered by around 30%, with no more than 1% overhead.

Critical strengths
  1. Reducing the memory usage without accuracy degradation. The lossless approach proposed preserves the uncompressed activation.

  2. Impressive optimization effort. Specifically, decoding achieves 97.4% of attainable bandwidth.

  3. Comprehensive evaluation. The baseline includes a wide range of existing works and consists of four nvCOMP codecs, CSR and CompAct. Additionally, both BF16 and FP8, two widely-received precisions, are tested in the experiments.

Critical weaknesses
  1. Inconsistency. The paper claims that BitSparse overhead is "no more than one per cent" in the Abstract and Conclusion, and "minimal overhead" in the Introduction. However, Table 7 results show that BitSparse (7.67s) requires more time than the baseline (7.42s) using the same micro-batch size. This result alone shows a three percent overhead.

  2. Lack of breakdown analysis. The paper is expected to be improved by a breakdown experiment, showing the ratios of encode, decode and fused backward in the step time.

  3. No multi-GPU experiments. Modern LLMs require multi-GPU training due to the model size and GPU memory capacity. Although BigSparse optimizes single-GPU memory usage, it can also reduce the data transfer size between GPUs in the multi-GPU scenario. Adding this would broaden this work.

Questions for authors
  1. What is the convergence under different micro-batch sizes (32 vs. 64)? In the full training comparison Section 6.5, the benefit of BitSparse is the low memory usage and thus larger micro batch size. Table 7 and Figure 6 in E.2. However, the convergence curves for micro-batch sizes 32 and 64 are not shown.

  2. What is the step time breakdown? Specifically, encode, decode, and fused backward, how much are they within a step?

  3. How will BitSparse perform in a larger LLM that requires multi-GPU? For example, a 70B parameter model?

2 comments

Thank you for taking the time to read our paper and recognising its strengths and potential improvements.

Inconsistency in overhead

The 1% overhead claim was specifically aimed at the Nemotron results which was the largest model tested. The NanoGPT model is smaller and has higher fixed overheads, which is why the overhead is higher. The abstract was very unclear in this regard, and we will fix it.

NanoGPT microbatch convergence

We reran the NanoGPT training with microbatch 32 and 64. Below are the results.

MethodMicrobatchFinal validation lossPeak GiBs/update
Dense323.27756814.9197.519
BitSparse-SB323.27838212.8657.649
BitSparse-SB643.27817821.2767.502

BitSparse-SB at microbatch 32 and 64 converge nearly identically. The slight differences in loss are caused by kernel non-determinism and gradient accumulation differences, since BitSparse's compression and decompression itself is lossless.

NanoGPT checkpoint timing and memory

Next we compared the speed and memory usage of dense, BitSparse and BitSparse-SB:

Methodtime/updateOverheadPeak GiB
Dense7.519 s—14.913
BitSparse7.541 s0.03%12.865
BitSparse-SB7.649 s1.44%12.805

BitSparse has negligable overhead, while BitSparse-SB has a small overhead of 1.73%. Memory usage is reduced by 13.7-14.1%.

NanoGPT phase timings

We separately measure the time taken for the forward, backward and optimizer phases. The results are shown below:

MethodForward msBackward msOptimizer msTotal ms
Dense2,561.04,905.5102.47,568.8
BitSparse2,591.54,872.7106.77,570.9
BitSparse-SB2,655.94,885.0107.97,648.7

The additional overhead of BitSparse-SB is primarily in the forward pass with the extra sign-bit packing. The backward pass is faster than the dense baseline thanks to our fused kernels. For further ablations into the effect of our tiled kernels, TensorBuffer and fused operators, see our response to reviewer 43ev. FP8 measurements are in our response to reviewer bYSo.

MultiGPU demonstration

BitSparse can be used to accelerate multi-GPU training by speeding up the transfer of activations between GPUs by compressing them. We demonstrate this with a simple 2-GPU experiment on a GH200 system connected by NVLink with a bandwidth of 125GB/s, transferring a 4 GiB activation tensor between GPUs. The results are shown below:

MethodSparsityTotal time / msEffective bandwidth / GB/sSpeedup
Dense transfer—34.32125—
BitSparse50%25.081711.37×
BitSparse90%10.694023.21×

BitSparse reduces the size of the tensors increasing the effective bandwidth. Timings include the overhead from compression and decompression so the scaling is not perfect. With 90% sparsity, BitSparse achieves a 3.21× speedup.

Unfortunately, a full comparison in a real multi-GPU training setup is beyond our hardware currently, which is why it isn't included in the paper, a substantial cluster would be required for a 70B model is effective with pipeline parallelism. However, the above results demonstrate that BitSparse can indeed be used to accelerate data transfer.

Reviewer_xCAWOpenReview ↗

Reviewer's response

Thank you for adding the experiment.

W1 and W2 are addressed. W3 is partially addressed, with a demo of two-GPU run on NanoGPT. Also, it is too early to claim scaling when only have one and two GPU runs.

Additionally, reviewer bYso reminds me that there are limited use case for the method.

I maintain my opinion on the score.

Reviewer_43evOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

BitSparse reduces the memory required to retain sparse feed-forward activations for backpropagation. It stores a packed support bitmap, compact nonzero values, and per-tile offsets, with optional removal of redundant sign bits. Custom Triton kernels compress and reconstruct the cache and fuse activation-gradient operations. A shared preallocated value buffer is intended to avoid synchronization when the compressed size is data-dependent. The method supports ReLU and squared ReLU, storing unsquared positive values for the latter, and preserves the selected BF16 or FP8 cache values rather than deliberately approximating them. The experiments include codec benchmarks, stacks of MLPs, forward/backward workloads for Modded-NanoGPT and Nemotron-H, and a complete Modded-NanoGPT training comparison. The reported memory-versus-token slopes decrease by about 31% and 30% for the two language models.

Critical strengths
  • The method exploits existing exact zeros without requiring a new training objective or sparsity-inducing approximation. In the real-model experiments, there is no reported activation pruning, magnitude thresholding, or additional objective forcing activations to zero for compression, which is good for a lossless compression method. ReLU itself produces the structural zeros, and BitSparse retains every nonzero value in the selected storage format.

  • The backward-information argument is well motivated. The paper identifies the positive preactivation values and positions needed for squared-ReLU gradients. Saving unsquared values avoids attempting to recover the original activation through a numerically problematic square root of rounded squared values.

  • The storage layout and accounting are explicit. The paper specifies tile geometry, bitmask size, prefix metadata, and optional sign removal with a theoretical compression upper bound so that the predicted compression ratios and break-even sparsity thresholds are understandable.

Critical weaknesses
  • The conceptual novelty over existing sparse activation storage is modest. First, prior work has already exploited ReLU sparsity, e.g., [ref1][ref2]. Bitmap-plus-nonzero encoding already appears in the cited prior work [ref3]. The most distinctive elements are implementation choices, but comparisons with an equivalently fused, modern bitmap baseline are limited. The pure-PyTorch comparison demonstrates the benefit of custom kernels, not necessarily a new storage or training principle.

  • Most large-model measurements are not full training steps. Appendix D.4 states that parameters are not updated, gradients are released as produced, and optimizer states and a retained full-model gradient buffer are absent. This is a valid controlled forward/backward workload, but it does not establish the same relative savings or throughput during full Nemotron training. Fixed parameter and optimizer storage materially reduce the percentage benefit of an activation-only optimization.

  • The practical usage of the proposed method is limited: 1) Modern LLMs rarely use ReLU but GELU and others. 2) Whether the fused kernel still has marginal overhead compared to other computations should be further verified with different models and hardware platforms.

[ref1] Speeding up convolutional neural networks by exploiting the sparsity of rectifier units, 2017. [ref2] Relu strikes back: Exploiting activation sparsity in large language models, 2024. [ref3] BitTrain: Sparse Bitmap Compression for Memory-Efficient Training on the Edge, 2021.

Questions for authors
  1. What is technically new beyond BitTrain? BitTrain already provides a precedent for bitmap-based sparse activation storage. Which aspects of BitSparse constitute a new systems-design contribution beyond adapting this representation to modern GPUs? Can you distinguish the benefits of tile organization, fused backward computation, sign-bit packing, and device-side buffer management through matched ablations?

  2. Do the large-model gains hold during complete training? The Nemotron-H measurements omit optimizer updates and persistent optimizer states and release gradients as they are produced. Can you provide a sustained training or fine-tuning measurement with optimizer updates, reporting total peak memory and throughput? If additional experiments are infeasible, what conclusions can reasonably be drawn from the current forward/backward-only results?

2 comments
AuthorsOpenReview ↗

Response to 43ev Part 1

Thank you for taking the time to read our paper and identifying strengths and areas for improvement.

Nemotron-H-8B full fine-tuning

In order to demonstrate sustained performance on a larger model, we fine-tuned Nemotron-H-8B on the FineWeb dataset for 2,000 updates / 32.8M tokens with sequence length 8,192 and batch size 2 on a GH200 system. The table below shows the peak memory, timings and final loss.

MethodPeak memoryTime/updateFinal validation loss
Dense77.806 GiB3.8593 s2.261730
BitSparse64.912 GiB3.8533 s2.261530
BitSparse-SB64.834 GiB3.8688 s2.261450

This includes the full training step, including the optimizer and parameter updates. Memory is reduced by 16.7% with essentially unchanged time, with a slowdown of 0.2%.

We also track the sparsity through training in new Figure 10. The average sparsity is around 92.7%, while the least sparse single batch is 91.8%. Sparsity and memory reduction remain consistent.

Comparison with existing methods

Many methods have been developed to exploit ReLU sparsity in inference [ref1][ref2]. However they are not applicable to training. These two methods exploit sparsity to avoid loading unneeded weights and remove unneeded multiplications / convolutions. This does not work in training when every neuron is likely activated once in a sequence / batch.

BitTrain [ref3] also uses a bitmask for ReLU sparsity. However, their method is limited compared to ours. BitTrain was developed for edge CPU devices with CNNs, and did not support GPU acceleration. Our method further introduces key features that are required for competitive modern LLM training. We run ablations to demonstrate.

Performance Ablations

First we compare BitSparse with: a flat Triton implementation (without our tiled layout) and pointwise fusion, Bitsparse without pointwise fusion and full implementations of Bitsparse(-sb). The pointwise fusion integrates the relu^2 activation and backwards sparse derivative into our kernels to reduce memory traffic. The time taken of the activation / compression for the forward (relu^2 activation and compression) and backward passes (decompression and relu^2 gradeint) for a Nemotron-H-8B layer is show below:

MethodForwardBackwardCombinedPeak allocated
Basic flat Triton, non-fused2.813 ms1.884 ms4.598 ms2,805 MiB
BitSparse raw, non-fused2.298 ms1.606 ms3.801 ms2,805 MiB
BitSparse raw, fused0.890 ms1.023 ms1.799 ms2,133 MiB
BitSparse-SB, fused1.050 ms1.125 ms2.039 ms2,240 MiB

The Naive version is over 2.5x slower than BitSparse, with higher transient memory usage as we materialise multiple intermediate tensors. The performance improvement is split between the tiled Triton implementation and fused pointwise operations. The fused version is 2.1x faster than the non-fused version, and 2.5x faster than the naive implementation.

Next we compare the effect of the improvements on a full Nemotron run, as well as the TensorBuffer.

BF16 methodPeak memoryTime/updateOverhead vs dense
Dense77.806 GiB3.8593 s—
Basic flat Triton, non-fused65.196 GiB3.9095 s1.30%
BitSparse, non-fused65.196 GiB3.8974 s0.99%
BitSparse, fused64.912 GiB3.8585 s−0.02%
BitSparse, fused + TensorBuffer64.912 GiB3.8533 s−0.16%
BitSparse-sb, fused + TensorBuffer64.834 GiB3.8688 s0.24%

BitSparse with the TensorBuffer is faster than the normal uncompressed implementation (similar to our synthetic MLP results). BitSparse-sb trades overhead for a further memory reduction. The non-fused and flat Triton implementations carry substantially more overhead.

AuthorsOpenReview ↗

Response to 43ev Part 2

Additional bandwidth measurements

On the GH200, we measure the bandwidth of the BitSparse codec with and without sign-bit packing. The results are shown below for sparsity rho=0.5:

Sign-bit packingTime / ms<br>EncodeTime / ms<br>DecodeTime / ms<br>CombinedBandwidth / TB/s<br>EncodeBandwidth / TB/s<br>DecodeBandwidth / TB/s<br>CombinedMemory BW / %<br>EncodeMemory BW / %<br>DecodeMemory BW / %<br>Combined
No2.9521.8784.7593.2292.7523.05688.575.483.8
Yes3.5193.1196.4882.9631.6262.38181.244.665.3

And for sparsity rho=0.8:

Sign-bit packingTime / ms<br>EncodeTime / ms<br>DecodeTime / ms<br>CombinedBandwidth / TB/s<br>EncodeBandwidth / TB/s<br>DecodeBandwidth / TB/s<br>CombinedMemory BW / %<br>EncodeMemory BW / %<br>DecodeMemory BW / %<br>Combined
No3.1661.9335.0493.1592.8053.05486.676.983.7
Yes4.1464.3288.3692.8141.2402.03577.134.055.8

Utilisation is relative to peak copy bandwidth of 3.65TB/s. These demonstrate that BitSparse generalises to a much more powerful GH200 GPU compared to our previous RTX 3090 results.

For additional FP8 training and GH200 codec measurements, see our response to reviewer [bYSo]. NanoGPT timing and convergence results are in our response to reviewer [xCAW].

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.