- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes BitSparse, a lossless activation compression method for LLM training with ReLU-family activations. It stores nonzero values together with masks and offsets, and implements efficient Triton kernels for compression and reconstruction. The method is evaluated on synthetic MLPs and two language models, with comparisons against several compression and checkpointing baselines.
Critical strengths
- Reducing activation memory is an important problem for efficient LLM training.
- BitSparse achieves substantial memory savings with little runtime overhead in the reported benchmarks.
- The source code is provided as supplementary material, supporting reproducibility.
Critical weaknesses
- The method is limited to ReLU-family activations with exact zeros, restricting its applicability to LLMs using other activations such as SwiGLU.
- The evaluation relies heavily on synthetic MLP benchmarks. End-to-end training is demonstrated only on a 162M model, while FP8 LLM benchmarks and end-to-end results for larger models (e.g., 8B) are missing.
- There are formatting and typographical issues. The "BitSparse:" prefix in the PDF title is missing from the OpenReview submission title. Table captions lack the required spacing below them. Line 320: "BisSparse" should be "BitSparse"; line 255: "there no" should be "there are no".
Questions for authors
- How is memory bandwidth utilization calculated in Table 2? The reported values at 0.9 sparsity seem to correspond to 0.8 sparsity instead.
- Why does the dense baseline's memory usage differ by approximately 1 GiB between 0.5 and 0.9 sparsity in Tables 3 and 8, whereas Tables 4 and 9-11 report identical values across sparsity levels?
- How is the preallocated TensorBuffer capacity determined? What happens if the number of nonzero activations exceeds its capacity during training?
2 comments
Thank you for the reply. Weakness 1 is unaddressed, and I think it is a fundamental weakness of this paper that cannot be resolved with a rebuttal. Reviewer 43ev also has this concern. I maintain my rating.
Thank you for taking the time to review our paper and check for issues and improvements.
Nemotron-H-8B FP8 fine-tuning
We compare Nemotron-H-8B full finetuning runs on a GH200 GPU for 2,000 updates / 32.8M tokens with sequence length 8,192 and batch size 2. The results in BF16 are given in our response to reviewer 43ev. Here, we give analagous results using FP8 activations and parameters with BF16 optimizer state. The GH200 supports FP8 better than our previously used RTX 3090.
Memory usage is reduced by 12.0–12.2% with a slowdown of 0.71–0.95%. The final validation loss is essentially unchanged up to 4 significant figures. Combined with the BF16 full finetuning results, this demonstrates that BitSparse is effective for real BF16 and FP8 fine-tuning.
Formatting corrections
Thank you for finding the formatting issues. We have corrected the title, table captions, and typos in the final version of the paper.
In Table 2, the bandwidth calculation did indeed use a sparsity of 0.8. Thank you for catching this issue.
In Table 3 and 8, the values at sparsity 0.9 are the correct values (11344 and 10039). This was a data entry error, and we will correct this in the final version of the paper. The percentage changes in the main text already reflected the correct values.
Size of the Tensor Buffer.
As we note in the paper, the sparsity is very consistent during training. Theoretically, the TensorBuffer can be directly sized with a small amount of overhead for the worst case sparsity by adding a few percent headroom to its size. A simpler approach is to make the TensorBuffer take up most of the spare memory after accounting for all other memory used by the model (parameters, gradients, optimiszer state etc) since they are always fixed. This gives close to the same size as if no TensorBuffer was used.
If the TensorBuffer runs out, BitSparse will stop with an error instead of causing numerical issues or truncating values.