- Rating
- 2
- Confidence
- 2(−1)
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces SLORR, an SVD-free framework that promotes low-rank weights during training through GPU-efficient approximations of spectral regularizers, without architectural changes, cached spectral states, or predefined target ranks. Instantiated with Hoyer spectral sparsity and nuclear norm objectives, SLORR achieves competitive compression–quality trade-offs across vision and language models with low training overhead, and improves post-compression performance over unregularized training at scales up to 1B parameters.
Critical strengths
- SLORR provides a simple, easily integrated framework for SVD-free spectral regularization without architectural changes or cached spectral states, supported by conditional approximation guarantees.
- Experiments span vision and language models, multiple compression procedures, and different optimizers, with direct measurements demonstrating low training overhead.
Critical weaknesses
- The language modeling experiments focus on pretraining from scratch, leaving applicability to existing pretrained LLMs insufficiently established. The authors also acknowledge difficulties in preserving general capabilities when applying low-rank regularization during LLM fine-tuning, which limits its practical scope.
- Comparisons with Q3R and LoRITa in language modeling are limited to 135M parameters. At 560M and 1B, comparisons primarily against unregularized training do not sufficiently establish SLORR’s compression–quality and training-cost advantages over existing methods at larger scales.
- The main experimental figures contain densely overlapping curves and small labels, making methods difficult to distinguish and performance differences at matched compression budgets difficult to assess. A concise table of representative operating points would improve readability.
Questions for authors
- Could you provide quantitative results from the exploratory fine-tuning experiments mentioned in the limitations?
- What prevents comparisons with Q3R and LoRITa at 560M and 1B scales? Could you compare their compression–quality trade-offs and training costs at one of these scales?
- How does the proposed method perform on larger or more recent LLMs?
4 comments
Response to Reviewer AAA9 (Part 2/2)
W3: Clarity of experimental figures
We thank you for bringing this up. We acknowledge the original figures were difficult to understand. We have made the figures larger and improved the style for better readability. Tables with the operating points are in the Appendix (Tables 15–18 for 135M and 560M, Table 19 for OLMo-2). We also reorganized the experimental section for better understanding.
Q3: Larger or more modern LLMs
Our new continued training experiments (see our answer to W1/Q1) are run at 1.5B scale and use another modern architecture (OLMo-2), further confirming the applicability of our method at larger scales and more modern architectures. Moreover, as shown in our answer to W2/Q2, our method remains very efficient at 3B scale, significantly more than the other baselines.
To further confirm the applicability to other modern architectures, we have also run new experiments on a more recent architecture, Qwen 3, at 135M scale and with a similar setup and sweeps as our other experiments. We find that SLORR still improves compressibility (Figure 9 in Section 4.4; all strengths in Appendix N.7).
Final remarks
We sincerely hope that we have addressed your concerns in the revised version, and would kindly ask you to update your score accordingly.
Thank you for the detailed responses and additional experiments, which have addressed most of my concerns. Some presentation and readability issues remain, particularly in the formatting of the equations and code listings. After considering the paper’s contributions, the additional evidence, and the remaining limitations, I am inclined to maintain my original score.
Thanks
Thank you for engaging with our response and taking a look at the revision, we really appreciate it. We are very happy to hear that your concerns are mostly resolved.
First, thanks for pointing out the formatting issue in Listing 1. We have fixed it (still in page 4) (we have made the font the same size as the main text and clearer and fixed the line breaks).
You also mention that other presentation/readability issues remain, particularly in the equations. Although we searched for such issues, we did not find anything nonstandard. However, we have updated the PDF and made sure that inline equations do not alter spacing in the text, we have numbered the displayed equations, and we have rewritten the display in Proposition 3.1 so that the bounds for SLORR-Nuc and SLORR-Hoyer read as two aligned lines (the result and content have not changed). We did not see any other issue but would be very happy to address any immediately if you can point it out concretely.
We also appreciate that you leave the assessment open even if you were inclined toward keeping your score. Given that your concerns had been mostly addressed in our original response as you mentioned, and that we have now fixed the remaining formatting issues, we'd be grateful if you could reconsider whether the paper is now above the bar. In any case, we respect your decision and thank you for taking your time to engage.
Response to Reviewer AAA9 (Part 1/2)
We thank the reviewer for appreciating the merits of our paper and for the interesting questions and suggestions. We address them below. Table and figure numbering refers to the revised manuscript.
Thanks for the interesting suggestion! As noted in our limitations paragraph, our point is that the training data is likely to be important. These experiments were very preliminary and used narrow fine-tuning data, so we do not have rigorous and reliable results to report for them. Instead, we have repeated them in a more careful setting and with appropriate data.
In particular, we take the public OLMo-2 1B checkpoint (about 1.5B parameters, trained for 4T tokens) and continue its training trajectory (which is publicly available) with regularization enabled for 20B tokens (0.5% of its pretraining). We compare a no regularization control (to verify that compressibility does not simply increase with more training) against Hoyer and Nuc. In fact, the unregularized control stays mostly the same, but the regularized models substantially improve at higher compression ratios. The following table shows the results (FineWeb PPL and average task accuracy) at 70% of the parameters (see Section 4.3 and Figure 8; full results in Table 19, Appendix N.5):
At 70% of the parameters, the regularized models keep almost all of their accuracy, while the original checkpoint and the control become nearly useless. We hope this clears up the concern about SLORR's practical scope.
Thank you for bringing this up. We agree comparisons at scale, both in terms of quality and speed, are important.
Training cost advantage. In our original submission, we already provided cost estimations. We have extended them and added them to the main text (Section 4.2, Figure 6; all numbers in Appendix O). For LLM training, the training time overhead and the added memory per GPU are:
We note that at 3B, LoRITa only fits at half the microbatch of the other baselines. In addition, on 80 GB GPUs, SLORR is the only method that fits our settings for 3B models (Appendix O).
Similarly, LoRITa adds 32–47% trainable parameters (Figure 6), whereas SLORR does not add any.
Quality comparisons at larger scales. These comparisons are expensive, since each baseline needs different hyperparameters, which is what prevented them in the submission. As you suggested, we have now run them at 560M. Interestingly, we find that the quality gap vs LoRITa and Q3R narrows when compared to our 135M results, and SLORR is even the best of the three at 30% of the parameters. The following table contains the Pareto frontier at 70%, 50% and 30% of the parameters, together with the training time overhead, the added memory per GPU and the additional trainable parameters of each method (full results in Figure 5 in the main text and Appendix N.2):
We note that we found higher LoRITa regularization (1.0, as in our 135M experiments) to diverge at 560M scale, something which we did not observe at the smaller 135M scale.
We have also revised some parts of the wording in the paper to avoid indicating that SLORR's objectives (Hoyer and Nuclear) are superior to the other baseline's objectives. Rather, we claim that the implementation is simpler, significantly more efficient and achieves competitive performance. This new 560M result indicates that they get closer at scale.