acceptodds
Under review as a conference paper at ICLR 2027

SLORR: Simple and Efficient In-Training Low-Rank Regularization

Abstract

Direct spectral regularization is a natural way to induce low-rank weights, but common objectives are costly at modern scale because they typically require repeated singular value decompositions (SVDs). Existing alternatives reduce this cost through reformulations that introduce architectural reparameterization, cached spectral state, predefined target ranks, or periodic decompositions, thus hindering practicality. We introduce SLORR to make direct spectral regularization practical without these restrictions while retaining competitive post-compression performance. Our SVD-free, stateless, architecture-preserving framework uses GPU-friendly approximations of the spectral quantities needed for regularizer forward and backward passes. SLORR supports different spectral objectives, instantiated here with Hoyer spectral sparsity and nuclear norm regularization. We evaluate induced compressibility and performance after post-training low-rank compression across different variants, using ImageNet-1K pretraining and continued training of ResNets and Vision Transformers, and LLM pretraining at 135M, 560M, and 1B parameters. Across vision settings and at 135M LLM scale, SLORR achieves competitive compression–quality tradeoffs with recent strong low-rank regularizers, often with greater robustness across settings, while retaining low training overhead and avoiding their limitations. We further scale SLORR to 560M and 1B LLMs: it substantially improves post-compression performance over matched unregularized training. These gains persist across different language model token budgets and four optimizers.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Low-Rank, Regularization, Compression

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_AAA9OpenReview ↗
Rating
2
Confidence
2(−1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces SLORR, an SVD-free framework that promotes low-rank weights during training through GPU-efficient approximations of spectral regularizers, without architectural changes, cached spectral states, or predefined target ranks. Instantiated with Hoyer spectral sparsity and nuclear norm objectives, SLORR achieves competitive compression–quality trade-offs across vision and language models with low training overhead, and improves post-compression performance over unregularized training at scales up to 1B parameters.

Critical strengths
  1. SLORR provides a simple, easily integrated framework for SVD-free spectral regularization without architectural changes or cached spectral states, supported by conditional approximation guarantees.
  2. Experiments span vision and language models, multiple compression procedures, and different optimizers, with direct measurements demonstrating low training overhead.
Critical weaknesses
  1. The language modeling experiments focus on pretraining from scratch, leaving applicability to existing pretrained LLMs insufficiently established. The authors also acknowledge difficulties in preserving general capabilities when applying low-rank regularization during LLM fine-tuning, which limits its practical scope.
  2. Comparisons with Q3R and LoRITa in language modeling are limited to 135M parameters. At 560M and 1B, comparisons primarily against unregularized training do not sufficiently establish SLORR’s compression–quality and training-cost advantages over existing methods at larger scales.
  3. The main experimental figures contain densely overlapping curves and small labels, making methods difficult to distinguish and performance differences at matched compression budgets difficult to assess. A concise table of representative operating points would improve readability.
Questions for authors
  1. Could you provide quantitative results from the exploratory fine-tuning experiments mentioned in the limitations?
  2. What prevents comparisons with Q3R and LoRITa at 560M and 1B scales? Could you compare their compression–quality trade-offs and training costs at one of these scales?
  3. How does the proposed method perform on larger or more recent LLMs?
4 comments
AuthorsOpenReview ↗

Response to Reviewer AAA9 (Part 1/2)

We thank the reviewer for appreciating the merits of our paper and for the interesting questions and suggestions. We address them below. Table and figure numbering refers to the revised manuscript.

W1/Q1: Experiments on LLMs beyond pretraining

Thanks for the interesting suggestion! As noted in our limitations paragraph, our point is that the training data is likely to be important. These experiments were very preliminary and used narrow fine-tuning data, so we do not have rigorous and reliable results to report for them. Instead, we have repeated them in a more careful setting and with appropriate data.

In particular, we take the public OLMo-2 1B checkpoint (about 1.5B parameters, trained for 4T tokens) and continue its training trajectory (which is publicly available) with regularization enabled for 20B tokens (0.5% of its pretraining). We compare a no regularization control (to verify that compressibility does not simply increase with more training) against Hoyer and Nuc. In fact, the unregularized control stays mostly the same, but the regularized models substantially improve at higher compression ratios. The following table shows the results (FineWeb PPL and average task accuracy) at 70% of the parameters (see Section 4.3 and Figure 8; full results in Table 19, Appendix N.5):

PPL / accuracy (%)70%, SVD-LLM70%, plain SVD
Public checkpoint42.5 / 36.52784 / 26.8
+20B tokens, no regularization43.1 / 36.22573 / 26.6
+20B tokens, SLORR-Hoyer18.2 / 48.718.3 / 49.8
+20B tokens, SLORR-Nuc19.8 / 49.520.6 / 50.7

At 70% of the parameters, the regularized models keep almost all of their accuracy, while the original checkpoint and the control become nearly useless. We hope this clears up the concern about SLORR's practical scope.

W2/Q2: Comparisons at scale

Thank you for bringing this up. We agree comparisons at scale, both in terms of quality and speed, are important.

Training cost advantage. In our original submission, we already provided cost estimations. We have extended them and added them to the main text (Section 4.2, Figure 6; all numbers in Appendix O). For LLM training, the training time overhead and the added memory per GPU are:

Training time / added memory (GiB)135M560M1B3B
SLORR+1.9% / 0.1+2.3% / 0.3+2.4% / 0.6+3.1% / 2.0
LoRITa+6.5% / 2.0+9.1% / 8.6+7.6% / 9.3+30.8% / 34.6
Q3R, refresh interval 300+8.4% / 0.8+8.4% / 4.1+8.6% / 7.8+10.6% / 24.8
Q3R, refresh interval 50+13.9% / 0.8+16.0% / 4.1+16.8% / 7.8+21.1% / 24.8
Q3R, refresh interval 5+74.0% / 0.8+98.0% / 4.1+105% / 7.8+135% / 24.8

We note that at 3B, LoRITa only fits at half the microbatch of the other baselines. In addition, on 80 GB GPUs, SLORR is the only method that fits our settings for 3B models (Appendix O).

Similarly, LoRITa adds 32–47% trainable parameters (Figure 6), whereas SLORR does not add any.

Quality comparisons at larger scales. These comparisons are expensive, since each baseline needs different hyperparameters, which is what prevented them in the submission. As you suggested, we have now run them at 560M. Interestingly, we find that the quality gap vs LoRITa and Q3R narrows when compared to our 135M results, and SLORR is even the best of the three at 30% of the parameters. The following table contains the Pareto frontier at 70%, 50% and 30% of the parameters, together with the training time overhead, the added memory per GPU and the additional trainable parameters of each method (full results in Figure 5 in the main text and Appendix N.2):

PPL (SVD-LLM / plain SVD)70%50%30%Training timeMemory (GiB)Additional params
No regularization18.2 / 28.636.6 / 581.2475.5 / 21,837–––
SLORR-Hoyer14.0 / 14.115.9 / 16.425.5 / 31.2+2.3%0.30%
SLORR-Nuc14.5 / 14.616.7 / 17.127.7 / 33.1+2.3%0.30%
Q3R13.8 / 13.814.7 / 15.236.7 / 178.2+8.4%4.10%
LoRITa13.5 / 13.714.8 / 15.630.0 / 47.3+9.1%8.6+42.3%

We note that we found higher LoRITa regularization (1.0, as in our 135M experiments) to diverge at 560M scale, something which we did not observe at the smaller 135M scale.

We have also revised some parts of the wording in the paper to avoid indicating that SLORR's objectives (Hoyer and Nuclear) are superior to the other baseline's objectives. Rather, we claim that the implementation is simpler, significantly more efficient and achieves competitive performance. This new 560M result indicates that they get closer at scale.

AuthorsOpenReview ↗

Response to Reviewer AAA9 (Part 2/2)

W3: Clarity of experimental figures

We thank you for bringing this up. We acknowledge the original figures were difficult to understand. We have made the figures larger and improved the style for better readability. Tables with the operating points are in the Appendix (Tables 15–18 for 135M and 560M, Table 19 for OLMo-2). We also reorganized the experimental section for better understanding.

Q3: Larger or more modern LLMs

Our new continued training experiments (see our answer to W1/Q1) are run at 1.5B scale and use another modern architecture (OLMo-2), further confirming the applicability of our method at larger scales and more modern architectures. Moreover, as shown in our answer to W2/Q2, our method remains very efficient at 3B scale, significantly more than the other baselines.

To further confirm the applicability to other modern architectures, we have also run new experiments on a more recent architecture, Qwen 3, at 135M scale and with a similar setup and sweeps as our other experiments. We find that SLORR still improves compressibility (Figure 9 in Section 4.4; all strengths in Appendix N.7).

Final remarks

We sincerely hope that we have addressed your concerns in the revised version, and would kindly ask you to update your score accordingly.

Thank you for the detailed responses and additional experiments, which have addressed most of my concerns. Some presentation and readability issues remain, particularly in the formatting of the equations and code listings. After considering the paper’s contributions, the additional evidence, and the remaining limitations, I am inclined to maintain my original score.

Thank you for engaging with our response and taking a look at the revision, we really appreciate it. We are very happy to hear that your concerns are mostly resolved.

First, thanks for pointing out the formatting issue in Listing 1. We have fixed it (still in page 4) (we have made the font the same size as the main text and clearer and fixed the line breaks).

You also mention that other presentation/readability issues remain, particularly in the equations. Although we searched for such issues, we did not find anything nonstandard. However, we have updated the PDF and made sure that inline equations do not alter spacing in the text, we have numbered the displayed equations, and we have rewritten the display in Proposition 3.1 so that the bounds for SLORR-Nuc and SLORR-Hoyer read as two aligned lines (the result and content have not changed). We did not see any other issue but would be very happy to address any immediately if you can point it out concretely.

We also appreciate that you leave the assessment open even if you were inclined toward keeping your score. Given that your concerns had been mostly addressed in our original response as you mentioned, and that we have now fixed the remaining formatting issues, we'd be grateful if you could reconsider whether the paper is now above the bar. In any case, we respect your decision and thank you for taking your time to engage.

Reviewer_sYa9OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

Enforcing sparsity in the singular values of neural network weight matrices is often desirable as it allows for reducing inference and deployment costs with no accuracy degradation. However, enforcing this during training is difficult as it typically requires either computing repeated SVDs or enforcing rank constraints prior to training. The authors of this work propose to instead regularize either the nuclear norm or the Hoyer metric. Normally, computing these would require computing and differentiating an SVD during every iteration. To avoid this overhead, the authors propose to use the polar decomposition. The authors apply their method to both vision and language model training and show that their method achieves results that are comparable to existing methods.

Critical strengths

The work presents a simple approach for computing the nuclear norm and it's corresponding gradient. While the paper focuses on using this to improve neural network compression, the technique is general and likely has further applications.

Critical weaknesses
  • The novelty of the work seems limited. The Hoyer and nuclear norm regularization terms are already well understood. The author's primary contribution is showing that these terms and their associated gradients can be approximated using the polar decomposition.
  • As written, the experimental results don't communicate the effectiveness of the proposed approach. Figure 1 is difficult to parse and Figure 3 seems to show worse performance than the baselines they compare with. From looking at the timing and memory overhead results in the appendix, I believe the proposed method is more scalable than the other approaches. However, the main paper needs to be able to effectively communicate this.
Questions for authors

Could the authors clarify the benefit of their method compared to other approaches?

5 comments
AuthorsOpenReview ↗

Response to Reviewer sYa9 (Part 1/2)

We thank the reviewer for appreciating the merits of our paper and for the interesting questions and suggestions. We address them below. Table and figure numbering refers to the revised manuscript, where new results are marked in blue.

W1: Clarification of novelty

We thank you for the comment and take this as an opportunity to clarify our contributions. As we point out in the paper, and you correctly mention in your review, the Hoyer and Nuclear objectives are known, and our core idea is that they and their gradients can be computed from an approximate polar factor. This is simple on purpose. All past work that avoids SVDs instead proposes more complicated solutions through proxy objectives that introduce new impracticalities (Table 1 in our submission). To the best of our knowledge, ours is the first work to directly regularize the intended objectives without SVDs (although the idea is simple, it has apparently been missed by the community, despite "SVD-free" regularization being a long-standing desire). Beyond the idea itself, we contribute an approximation guarantee for both the value and the gradient of each variant (Proposition 3.1), evidence that training behaves as with exact SVDs (Section 4.1), and, to our knowledge, the first study of this at scale (LLMs of up to 1.5B parameters).

W2: Clarity of the experimental results

We thank you for the honest comments. We acknowledge that the experimental section of the submission was hard to follow, and we have revisited it based on your comments.

Figure 1. We have made Figure 1 of the submission (Figure 4 in the revision) larger and used a clearer style that highlights the Pareto frontier. It shows that the Hoyer and Nuclear objectives, when used with SLORR, are competitive with Q3R and LoRITa across the vision models.

Figure 3. You are right on the point that the objectives our framework enables do not necessarily surpass those of Q3R and LoRITa in that setting. Our goal is not to claim that Nuclear norm and Hoyer regularization always outperform these, but to show that the more classic objectives can be computed efficiently. We have made this clearer in the revision. We now also compare against exact SVD-based regularization (Section 4.1, Figure 2): training with exact SVDs does not change the results meaningfully. We take this to mean that, in the settings where there is a gap (both the ones in which SLORR is better and the ones where it is worse) it is due to the objective and not to our approximation.

We have also added the same comparison for Llama-560M. Interestingly, we find that the quality gap vs LoRITa and Q3R narrows when compared to our 135M results, and SLORR is even the best of the three at 30% of the parameters. The following table contains the Pareto frontier at 70%, 50% and 30% of the parameters, together with the training time overhead, the added memory per GPU and the additional trainable parameters of each method (full results in Figure 5 in the main text and Appendix N.2):

PPL (SVD-LLM / plain SVD)70%50%30%Training timeMemory (GiB)Additional params
No regularization18.2 / 28.636.6 / 581.2475.5 / 21,837–––
SLORR-Hoyer14.0 / 14.115.9 / 16.425.5 / 31.2+2.3%0.30%
SLORR-Nuc14.5 / 14.616.7 / 17.127.7 / 33.1+2.3%0.30%
Q3R13.8 / 13.814.7 / 15.236.7 / 178.2+8.4%4.10%
LoRITa13.5 / 13.714.8 / 15.630.0 / 47.3+9.1%8.6+42.3%

Efficiency and scalability. We also thank you for digging deeper and highlighting our efficiency and scalability results. These are one of our main points, and we agree they were not well presented. We have extended them and moved them from the appendix to the main text (Section 4.2, Figure 6): SLORR has the lowest runtime and memory overhead at every LLM scale we measured, from 135M to 3B. The following table gives the training time overhead and the added memory per GPU (the entire set of results and settings is in Appendix O):

Training time / added memory (GiB)135M560M1B3B
SLORR+1.9% / 0.1+2.3% / 0.3+2.4% / 0.6+3.1% / 2.0
LoRITa+6.5% / 2.0+9.1% / 8.6+7.6% / 9.3+30.8% / 34.6
Q3R, refresh interval 300+8.4% / 0.8+8.4% / 4.1+8.6% / 7.8+10.6% / 24.8
Q3R, refresh interval 50+13.9% / 0.8+16.0% / 4.1+16.8% / 7.8+21.1% / 24.8
Q3R, refresh interval 5+74.0% / 0.8+98.0% / 4.1+105% / 7.8+135% / 24.8

We note that at 3B, LoRITa only fits at half the microbatch of the other baselines. In addition, on 80 GB GPUs, SLORR is the only method that fits our settings for 3B models (Appendix O).

Similarly, LoRITa adds 32–47% trainable parameters (Figure 6), whereas SLORR does not add any.

AuthorsOpenReview ↗

Response to Reviewer sYa9 (Part 2/2)

Improved structure and presentation of experiments. Following your comments, we have restructured the presentation of our experiments. The experimental section now reads in this order: analysis of the approximation against exact SVDs (Section 4.1, Figures 2–3), the comparison against the other objectives and their cost (Section 4.2, Figures 4–6), scaling (Section 4.3, Figures 7–8), and robustness across settings (Section 4.4, Figure 9). Each section now also has a small box with the main takeaways. We hope this makes it easier to follow and understand our main claims.

Q1: Benefits compared to other approaches

We thank the reviewer for the opportunity to clarify this point! We remark the benefits of our framework:

  • All past objectives either modify the architecture (LoRITa adds 32–47% trainable parameters in our LLM settings, Figure 6), rely on assumptions such as the validity of cached quantities and state, or introduce additional hyperparameters like exact target ranks. Q3R still relies on periodic SVDs. SLORR avoids all these and its operations are all GPU-friendly. It is also simpler to implement and integrate. This is showcased in Table 1.
  • SLORR scales better than other approaches in our settings (second table above; Figure 6 in the revision). It is also the only approach that regularizes the classic objectives rather than proxies.
  • As you affirm in your review (thanks for highlighting this point!), avoiding these limitations enables SLORR to work on pretty much every matrix one would want to regularize. In particular, given that it only needs the matrix at the current step without any other assumptions, it can be applied to matrices that are not weights and change at every step, such as activations, where cached state or a reparameterization do not apply directly. This likely enables future applications, as you pointed out.

Final remarks

We sincerely hope that we have addressed your concerns in the revised version, and would kindly ask you to update your score accordingly.

AuthorsOpenReview ↗

Kind reminder

Dear Reviewer sYa9,

As the discussion period ends in less than a day, we wanted to kindly check whether our response and the revised paper address your concerns. We would be grateful if you could take our clarifications and improvements into account in your final assessment, and we are more than happy to answer any remaining question before the deadline.

Reviewer_MmgYOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes SLORR, a method for low-rank regularization during training without computing an SVD at every step. The authors use Polar Express to approximate the polar factor UV⊤UV^\top and then use it to compute the nuclear-norm and Hoyer regularizers and their gradients. Unlike LoRITa, the method does not change the model parameterization; unlike Q3R, it does not need cached SVDs or a target rank. SLORR improves compressibility compared with training without regularization, and the reported training overhead is small. Its advantage in compression quality over other regularizers is less clear.

Critical strengths
  1. The paper is easy to follow, and the method is simple to implement. It works directly with the original weight matrices and does not require additional trainable factors or periodic SVDs.

  2. The experimental study is quite broad. The authors test several vision models and LLM sizes, two compression procedures, different optimizers, and different training budgets.

  3. I find the efficiency results useful. The authors report both runtime and memory, and SLORR has low overhead even for larger models. The implementation details and code also make the results easier to reproduce.

Critical weaknesses
  1. It is hard to distinguish the authors contribution from existing work. Both regularizers are known, the formulas involving the polar factor are standard, and Polar Express is also an existing method. As I understand it, the main novelty is using Polar Express to compute the regularizer and its gradient during training. I would like the authors to make this distinction clearer and explicit. Also, the paper calls SLORR a general framework, but only considers two objectives that both depend on the nuclear and Frobenius norms.

  2. The theoretical approximation bound seems to be loose for the actual setup. For example, with ℓ=10−3\ell=10^{-3}, degree d=5d=5, and six iterations, Proposition 3.1 gives probability very close to 1. Therefore, it is hard to understand what the guarantee tells us about the approximation used in the experiments. This does not mean that the approximation is inaccurate, but I would like to see the actual error in the regularizer and its gradient compared with exact SVD. The assumed lower bound on nonzero singular values is also questionable when the method is designed to make the spectrum decay faster.

  3. At 560M and 1B scale, the quality results compare SLORR mainly with no regularization. The comparisons with Q3R and LoRITa are limited to 135M, where SLORR is competitive but not always better. The memory and runtime results explain why larger comparisons are difficult, but they do not establish that SLORR has a better compression-quality tradeoff than the alternatives at these scales.

Questions for authors
  1. How large is the error in the regularizer values and gradients compared with exact SVD during training, especially for layers with many small singular values?

  2. What other spectral regularizers can be implemented in the same way, beyond functions of the nuclear and Frobenius norms?

  3. How does SLORR compare with Q3R when the refresh interval is tuned and the total training time is taken into account?

  4. How stable are the main LLM results across different random seeds? Even a few repeated configurations would help.

4 comments
AuthorsOpenReview ↗

Response to Reviewer MmgY (Part 1/2)

We thank the reviewer for appreciating the merits of our paper and for the interesting questions and suggestions. We address them below. Table and figure numbering refers to the revised manuscript, where new results are marked in blue.

W1: Clarification of contributions

Thanks for the suggestion! We agree on this point. The goal of our work is not to introduce a new objective but to make the most popular ones usable. Past work instead relies on proxy objectives, and to the best of our knowledge, we are the first to directly regularize the intended objectives without SVDs. As you say, the regularizers, the polar factor formulas and Polar Express are known. What is novel (and has apparently been missed by the community, despite "SVD-free" regularization being a long-standing desire) is combining them into an SVD-free regularizer usable during training, with an approximation guarantee for both its value and its gradient (Proposition 3.1), evidence that training behaves as with exact SVDs (Section 4.1), and the study at scale. We have made this clearer in the revision. To our knowledge, our work is also the first to scale direct rank regularization to LLMs (of up to 1.5B parameters). We discuss the "general framework" question in our answer to Q2.

W2/Q1: Theoretical bounds and approximation error

Thanks for bringing this important point to our attention.

Theoretical and numerical error. You are right about the bound. We clarify that the bound is not probabilistic. It bounds the worst-case scenario. Our main goal with the result was to show that it converges to the true objective.

You are also right in your prediction that, in practice, the error is small with our settings. We agree that this is an important point and we have clarified it in the revision: we have added a plot near the discussion (Figure 1) that compares (1) the worst-case bound predicted by the theoretical analysis, (2) value error and (3) weighted gradient alignment. The matrix is representative (taken from a regularized model at the end of training) and a "hard" case: 48% of its normalized singular values are below 10−310^{-3}, the situation you pointed out. Further details are in Appendix E.1.

The bound value reduces drastically after some more iterations (14+). At 6 iterations, in practice, the value and gradients have already mostly converged. This coincides with our training ablation already present in the submission, which we have moved to the main text (Figure 3) for clarity.

Impact of approximations on training. We have also added experiments to compare the impact of approximating the regularizer against exact SVDs for entire training trajectories. We repeated some ResNet-50 and 135M LLM experiments for both regularizer variants (please, see Figure 2 in Section 4.1 in the revision for more details):

  • We plot the spectra of final models. We find that both exact SVD and SLORR achieve similar patterns.
  • We also evaluate compressibility. Our approximation does not change the results meaningfully with respect to training with exact SVDs (all numbers in Table 14).

Lower bound. Your observation is correct. However, as pointed out by the Polar Express authors and discussed in our submission, in theory, this simply results in slower convergence for the singular values below the bound. In practice, this only affects singular values that are already close to zero, and training with exact SVDs gives closely matching models (Section 4.1). The lower bound, which we set to 10−310^{-3}, is also the point at which precision (bf16 in the Polar Express iterations) comes into play rather than the bound. We, again, find this not to be important in practice in the experiments against exact SVD.

W3: Comparison against other regularizers

As you point out that the quality over other regularizers is less clear, we first want to clarify the objective of our paper, which is to make the more classical regularizers usable efficiently at large scale. SVD-based regularization closely following our approximation along entire training trajectories suggests that the inter-regularizer differences are not due to our approximation but because of other factors. For example, LoRITa adds 32–47% trainable parameters in our LLM settings (Figure 6), and Q3R, through expensive machinery, uses a specific target rank. We avoid claiming that Nuclear or Hoyer-based regularization outperforms these, and have made this clearer in the paper.

AuthorsOpenReview ↗

Response to Reviewer MmgY (Part 2/2)

Comparisons at larger scales. Nevertheless, we have included comparisons at 560M scale. Interestingly, we find that the quality gap vs LoRITa and Q3R narrows vs our 135M experiments, and SLORR is even the best of the three at 30% of the parameters. The following table contains the Pareto frontier at 70%, 50% and 30% of the parameters, together with the training time overhead, the added memory per GPU and the additional trainable parameters of each method (full results in Figure 5 in the main text and Appendix N.2):

PPL (SVD-LLM / plain SVD)70%50%30%Training timeMemory (GiB)Additional params
No regularization18.2 / 28.636.6 / 581.2475.5 / 21,837–––
SLORR-Hoyer14.0 / 14.115.9 / 16.425.5 / 31.2+2.3%0.30%
SLORR-Nuc14.5 / 14.616.7 / 17.127.7 / 33.1+2.3%0.30%
Q3R13.8 / 13.814.7 / 15.236.7 / 178.2+8.4%4.10%
LoRITa13.5 / 13.714.8 / 15.630.0 / 47.3+9.1%8.6+42.3%

We note that we found higher LoRITa regularization (1.0, as in our 135M experiments) to diverge at 560M scale.

We have also revised some parts of the wording in the paper to avoid indicating that SLORR's objectives (Hoyer and Nuclear) are superior to the other baseline's objectives. Rather, we claim that the implementation is simpler, significantly more efficient and achieves competitive performance. This new 560M result indicates that they get closer at scale.

Q2: Generality of our framework

Thanks for this question. By general we mean, precisely, any function f(W,∥W∥∗)f(W, \Vert W\Vert_{\ast}) where ∇Wf\nabla_W f can be computed efficiently once ∥W∥∗\Vert W\Vert_{\ast} is known. Examples beyond our two instantiations are the nuclear norm minus the Frobenius norm, ∥W∥∗−∥W∥F\Vert W\Vert_{\ast} - \Vert W\Vert_F, or the ratio of the nuclear norm to the spectral norm, ∥W∥∗/∥W∥2\Vert W\Vert_{\ast} / \Vert W\Vert_2, where the spectral norm and its gradient are cheap to obtain with a few power iterations. We agree the current wording is not clear and have fixed this in the revision. We have also included a short paragraph towards the end of Section 3, where we discuss this point.

Q3: Sweeping Q3R's refresh factor

Thanks for the interesting suggestion!

Tuning Q3R's refresh interval. The objective function of Q3R is a smoothed log-determinant. This, when done exactly, requires an SVD at every iteration. The refresh interval of Q3R simply determines how good the approximation is (lower interval = better approximation), so we do not consider it a crucial factor to sweep for quality reasons. Moreover, jointly tuning 3 hyperparameters would be infeasible. We note, however, that our 135M LLM experiments already use different values (5 and 300). Similarly, we have already conducted an ablation in a subset of our settings in Appendix G.

Total training time. Thanks for this question. We first summarize the LLM training-time overheads and the added memory per GPU (you can find the entire set of results and settings in Section 4.2 and Appendix O):

Training time / added memory (GiB)135M560M1B3B
SLORR+1.9% / 0.1+2.3% / 0.3+2.4% / 0.6+3.1% / 2.0
LoRITa+6.5% / 2.0+9.1% / 8.6+7.6% / 9.3+30.8% / 34.6
Q3R, refresh interval 300+8.4% / 0.8+8.4% / 4.1+8.6% / 7.8+10.6% / 24.8
Q3R, refresh interval 50+13.9% / 0.8+16.0% / 4.1+16.8% / 7.8+21.1% / 24.8
Q3R, refresh interval 5+74.0% / 0.8+98.0% / 4.1+105% / 7.8+135% / 24.8

We note that, at 3B, LoRITa only fits at half the microbatch of the other baselines.

Although we did consider it when preparing our paper, we did not compare at matched wall-clock time, as it would introduce additional factors to the comparison that could confound results, such as the learning rate or its schedule, and could possibly lead to misleading comparisons. We note, however, that in pretraining seeing more tokens is expected to result in SLORR outperforming the baselines.

Q4: More seeds

We thank you for the suggestion. Our original submission already included random seed results for ResNet-50, finding that the variability is low (Appendix H.1). Following your request, we have also added a comparison for an LLM setting, Llama-135M with SLORR-Hoyer and three seeds (Table 20 in Appendix N.6):

PPL (SVD-LLM)70%50%40%
Seed 124.0025.4027.87
Seed 223.9525.6028.19
Seed 324.1725.5328.20
No regularization29.0052.7590.66

We again find that the variability between seeds is small.

Final remarks

We sincerely hope that we have addressed your concerns in the revised version, and would kindly ask you to update your score accordingly.

Thanks to the authors for clarification. My main concerns are now addressed, so I will raise my score from 2 to 3.

Thank you for your suggestions, which helped strengthen our paper and make it clearer. We are very happy to hear that your main concerns are addressed, and we further thank you for raising the score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response to all reviewers

We would like to thank all three reviewers for their careful reading of our paper. We are glad that they agree on its main merits:

  • Simplicity. All three reviewers describe the method as simple: "simple to implement" (MmgY), "a simple approach" (sYa9), and "a simple, easily integrated framework for SVD-free spectral regularization without architectural changes or cached spectral states" (AAA9).
  • Generality and soundness. The technique "is general and likely has further applications" (sYa9), and is "supported by conditional approximation guarantees" (AAA9).
  • Breadth of the experiments. The experimental study is "quite broad" (MmgY), spanning "vision and language models, multiple compression procedures, and different optimizers" (AAA9).
  • Efficiency. The efficiency results are "useful" (MmgY), with "direct measurements demonstrating low training overhead" (AAA9).

In short, SLORR makes the classic low-rank regularization objectives usable during training at scale: it achieves compression quality in the same range as recent proxy-based methods, at a fraction of their cost and without their restrictions.

The constructive suggestions allowed us to improve the paper, which we hope the reviewers will find to satisfactorily address their concerns and questions. We have uploaded a revised version of the paper incorporating the reviewers' suggestions. New results and discussion are highlighted in blue. Following the comments of Reviewer sYa9, we have reorganized the experimental section for readability. Its text is only marked in blue when the content is new (some parts have been compressed or slightly altered, we do not mark these to allow the reviewers to clearly see the new additions; note that this only applies to Section 4). Unless stated otherwise, all references refer to those in the revised version. We answer each reviewer's specific points in the corresponding responses.

We hope that, based on the answers to their questions and the improvements made in the revised version, the reviewers will reconsider their scores.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.