- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes S-SALAAD, a training framework that learns structured sparse-plus-low-rank representations for language-model weights. It extends SALAAD by introducing hardware-friendly sparsity patterns, aiming to turn parameter savings into faster inference. An iterative SVD reduces the cost of learning the low-rank components, while a budget-aware refinement procedure compresses one checkpoint to different model sizes without calibration data. Dedicated CPU and GPU kernels execute the compressed representations. Experiments on Llama pretraining and Qwen3 distillation show improved tradeoffs between model quality, memory usage, and prefill throughput compared with the evaluated baselines.
Critical strengths
1. The paper addresses a practical problem: reducing model size does not necessarily make inference faster. Learning hardware-friendly sparsity during training is well motivated, and the custom kernels help translate the learned structure into measured inference speedups.
2. The implementation goes beyond reporting theoretical parameter savings. The custom kernels demonstrate measured acceleration, and the inference-memory measurements include sparse indices and working buffers. The comparison with exact SVD also supports the use of the cheaper iterative update, which substantially reduces training overhead while preserving compression quality.
Critical weaknesses
1. The inference evaluation mainly measures throughput with relatively large batches. Since the paper targets edge deployment, it is unclear whether the reported gains also hold for batch-size-one inference and token-by-token generation. Quantized baselines are also missing, so the reported gains do not establish an advantage over a common alternative under similar memory constraints.
2. The choice and sensitivity of the target rank ratio and sparsity level are insufficiently investigated. Although the selected values are reported, it is unclear how much tuning is required for a new model or whether small changes substantially affect compression quality. This limits confidence that the reported tradeoffs can be reproduced beyond the evaluated configurations.
Questions for authors
1. Does S-SALAAD retain its speed advantage at batch size one and during autoregressive generation? Could you report inference speed and peak memory in these settings?
2. How does S-SALAAD compare with 8-bit or 4-bit quantization under similar memory budgets? Can the two approaches be combined to further reduce memory usage?
3. How were the target rank ratio and sparsity level selected? Does changing these values substantially affect the accuracy of the compressed models?
4. The evaluation includes a recent model architecture, but remains limited to relatively small models. Does S-SALAAD retain its compression quality and inference advantage on models with 7B or more parameters?
3 comments
Follow-up
Thank you again for your valuable feedback. Have our responses addressed your concerns? We would be happy to provide further clarification and look forward to hearing your thoughts!
I thank the authors for their detailed response and the additional experiments. The rebuttal resolves most of my concerns, and I will maintain my positive assessment.
New Experiments on Autoregressive Decoding, Quantization, and Target Sensitivity
Dear Reviewer Fw91,
Thank you for your careful review and constructive feedback. We respond to each weakness and question below.
1. Batch-size-one autoregressive generation (Weakness 1 and Question 1)
We agree that batch-size-one autoregressive generation is important for edge deployment. We have added experiments on Llama-1B and Qwen3-1.7B across Quadro RTX 6000, RTX 2080 Ti, and AMD EPYC 7662 CPUs, evaluating decode latency and peak memory across parameter budgets.
We highlight two representative results below; the complete results are available in Appendix H, Table 15. Latency and peak memory are shown as dense to S-SALAAD; memory reduction is relative to dense inference.
Compared with batched prefill, speedups are generally smaller, consistent with reduced weight reuse and lower Tensor Core utilization during single-token decoding. Relative memory savings are greater because activation storage is smaller. Some GPU configurations at larger budgets are slightly slower than dense while still reducing peak memory.
2. Comparison and combination with quantization (Weakness 1 and Question 2)
S-SALAAD induces compressible structures during pretraining and can complement post-hoc methods. Section 5.1 and Appendix G.1 demonstrate synergies with SliceGPT and SVD-LLM on its L-only checkpoints. Our new results also demonstrate compatibility with quantization.
We added W8A8 and W4A4 experiments on Llama-1B and Qwen3-1.7B (Appendix F, Tables 12–14). W8A8 causes only small perplexity increases, as for dense baselines. The table highlights Qwen3-1.7B at a 1B S-SALAAD budget on Quadro RTX 6000 during batched inference.
At similar peak memory, dense W8A8 achieves lower perplexity than unquantized S-SALAAD, with similar throughput. Combining W8A8 with S-SALAAD further reduces its peak memory by 17.0% and increases throughput by 19.7%, with only a 0.02 perplexity increase. The full results show additional throughput and memory benefits across all three platforms, supporting the complementarity of learned compressibility and quantization.
3. Target selection and sensitivity (Weakness 2 and Question 3)
Our I-Controller dynamically updates the regularization strengths α and β to regulate rank and sparsity (Appendix C.1, Eq. 16). These updates depend jointly on the target errors and controller step sizes Δα and Δβ. To simplify tuning, we fix the empirical targets (Γ^,Υ^) at (0.15, 0.95) in the main block-sparse experiments and adjust the step sizes. These targets encourage a low rank ratio and high sparsity during training.
Appendix D.3 already explains that structured sparsity uses a larger Δβ and smaller Δα than the unstructured baseline. Table 6 lists the values, which can generally be reused across model scales within each sparsity pattern.
We have added a sensitivity study in Appendix E.5 on Llama-350M with 16×16 and 32×32 blocks, comparing the default target pair (0.15, 0.95) with (0.10, 0.97), (0.20, 0.93), and (0.00, 1.00). The results support robustness to target choices. Figure 23 shows broadly consistent convergence trends across the layers and block sizes, and Table 11 shows small differences in final structural statistics between the default and nearby targets. Figure 24 shows limited changes in perplexity at larger budgets, with pronounced differences mainly under extreme compression.
Therefore, within the tested range, S-SALAAD is relatively insensitive to target choices at larger parameter budgets, and the shared empirical defaults can be reused without fine-grained target tuning. Under extreme compression, target selection has a larger effect on quality, and additional tuning may be beneficial.
4. Extension to models with 7B or more parameters (Question 4)
We agree that validation on larger models would strengthen the evaluation. Our current computational budget limits our experiments to models up to Qwen3-1.7B. We are actively seeking additional computational resources to extend these experiments to 7B and larger models and validate our findings on compression quality and inference efficiency at these scales.
The corresponding textual additions are marked in red in the revised manuscript. Thank you again for helping us improve the evaluation. If these additional results and clarifications address your concerns, we would sincerely appreciate your consideration of a higher rating.
Best,
The Authors