- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The authors study the issues of scalings for the two factors in LoRA, specifically for the non-zero structured initializer case. They propose SignSGD and other variant to improve the optimization for structured initializers without changing the actual initialization.
Critical strengths
The work seemingly unify or summarize a way to view the many previous works in the literature.
Critical weaknesses
-
One the main issue is writing. The equations, definition, and descriptions are scatter around the places. It is hard to track what are the proposals and the reasoning behind them. For example. in Sec 4, the readers are pointed to equation 7 and appendix N respectively to see the definition of the method and some of the property it has without any explanation.
-
The empirical evaluations seem incomplete and the relationship with some related work remains unclear. Methods like LoRA-RITE and LoRA-Muon are mentioned, but seemingly never compared in the experiments, while these method seem to be applicable.
Questions for authors
Could the authors compare with LoRA-RITE and LoRA-Muon, or explainable why they are not comparable?
4 comments
Response to Reviewer mtWi [Part II]
W2: "The empirical evaluations seem incomplete and the relationship with some related work remains unclear."
R4: Our method changes the factor scales once, before training. The training optimizer stays unchanged. LoRA-RITE and LoRA-Muon address factor-aware updates throughout training. We choose initial scales automatically, then use AdamW. Part I shows the practical value of this change before training.
The controls in Table 1 help explain this benefit. Configurations match Part I. The five candidates are five preset combinations of the common learning rate and the factor learning-rate ratio, as specified in Appendix G. Scores are percentages averaged over five seeds.
| Model | Method | GSM8K | MATH-500 | MMLU-Pro | BBH | Avg. |
|---|---|---|---|---|---|---|
| 0.6B | Spectrum-full | 56.74 | 23.56 | 23.10 | 33.73 | 34.28 |
| 0.6B | Tuned dual learning rates, 5 candidates | 56.51 | 23.44 | 23.02 | 33.61 | 34.15 |
| 0.6B | Layer-norm match | 56.29 | 22.68 | 22.90 | 33.13 | 33.75 |
| 0.6B | Warm | 56.82 | 23.80 | 23.44 | 33.90 | 34.49 |
| 0.6B | Warm-mapped learning rates | 56.92 | 23.80 | 23.46 | 33.78 | 34.49 |
| 1.7B | Spectrum-full | 68.32 | 35.32 | 35.60 | 43.16 | 45.60 |
| 1.7B | Tuned dual learning rates, 5 candidates | 67.99 | 35.24 | 35.60 | 43.04 | 45.47 |
| 1.7B | Layer-norm match | 67.95 | 35.04 | 35.56 | 42.64 | 45.30 |
| 1.7B | Warm | 68.32 | 35.80 | 35.85 | 43.08 | 45.76 |
| 1.7B | Warm-mapped learning rates | 68.31 | 35.88 | 35.79 | 43.10 | 45.77 |
We determine the ratio automatically. Matching update size alone gives less benefit, while learning rates mapped from Warm recover its scores. Section 5.2 now explains that the chosen allocation matters; Section 1 identifies automatic selection as the shared benefit of both implementations.
We also tested how the methods handle changes in the initial factor scales. Direct makes AdamW much less sensitive to the initial scale in this test. On Qwen3-1.7B/MiLoRA, rank 64, we apply , with . Each representation has the same initial adapter, frozen residual, and output.
Each method keeps the rate selected at . Direct recalibrates once at each starting scale, without retuning. Scores are five-seed means. The rows reproduce Part I.
| Method | Initial scale | GSM8K | MATH-500 | MMLU-Pro | BBH | Avg. |
|---|---|---|---|---|---|---|
| Raw + AdamW | 67.28 | 33.56 | 34.97 | 42.46 | 44.57 | |
| Raw + AdamW | 67.66 | 34.56 | 35.33 | 42.69 | 45.06 | |
| Raw + AdamW | 67.73 | 34.16 | 35.15 | 42.45 | 44.87 | |
| Direct + AdamW | 68.29 | 35.20 | 35.58 | 43.64 | 45.68 | |
| Direct + AdamW | 68.28 | 35.08 | 35.82 | 43.28 | 45.62 | |
| Direct + AdamW | 68.26 | 34.80 | 35.65 | 43.30 | 45.50 | |
| Raw + LoRA-RITE | 67.41 | 36.16 | 35.85 | 42.97 | 45.60 | |
| Raw + LoRA-RITE | 67.76 | 36.44 | 35.74 | 43.02 | 45.74 | |
| Raw + LoRA-RITE | 67.54 | 35.76 | 35.37 | 42.72 | 45.35 | |
| Raw + LoRA-Muon | 68.76 | 35.04 | 35.45 | 43.19 | 45.61 | |
| Raw + LoRA-Muon | 68.95 | 34.96 | 35.88 | 43.67 | 45.86 | |
| Raw + LoRA-Muon | 68.28 | 34.52 | 36.11 | 43.16 | 45.52 |
Direct reduces AdamW's sensitivity to the starting scale without changing the training optimizer. Its average-score range is 0.18 points, compared with 0.49 for Raw AdamW, 0.39 for LoRA-RITE, and 0.34 for LoRA-Muon. Section 5.4 now highlights this benefit.
The original study improves all nine model-initializer averages across Qwen and Llama. Complete Warm calibration adds 0.42% time in the reported configuration, with no change to inference or trainable parameters.
Revision locations Section 5.4 and Table 4 present the optimizer comparison; Appendix P.6, Table 39, gives the scale test. Section 4 explains the procedure before its derivation.
Thank you again for your valuable feedback, which has greatly improved our work. We hope these responses address your concerns. Please let us know of any further questions.
The revision seems to improve the writing. Currently, I am more or less neutral about the acceptance (which is slightly more positive than original).
Dear Reviewer mtWi,
Thank you very much for taking another look at our revision and for your encouraging feedback. We sincerely appreciate your recognition of the improvements.
We are glad that the revised presentation is clearer. We also hope that the newly added comparisons with LoRA-RITE and LoRA-Muon, together with the scale-sensitivity experiments, help address your original concerns about empirical completeness and the relationship to prior work.
We are grateful that these changes have moved your assessment in a more positive direction, and we hope they can be reflected in your final evaluation.
Thank you again for your time and constructive feedback.
Response to Reviewer mtWi [Part I]
Hi, Reviewer mtWi:
Thank you for your careful reading and detailed suggestions. We sincerely appreciate the time you spent following the method. Your comments pinpoint where the presentation needs a clearer explanation. We have revised the method and expanded the optimizer comparisons.
Q1: "The equations, definition, and descriptions are scatter around the places."R1: Thank you for this careful feedback and the specific example in Section 4. We apologize for making the method harder to follow than it needs to be. We agree that the presentation needs a broader revision. We have placed each scale rule beside its purpose and conditions before referring readers elsewhere.
Beyond this example, we have reviewed the equations, definitions, and cross-references throughout the paper. We have clarified symbol definitions, placed rules beside their explanations, and removed unnecessary jumps between sections.
For Equation 7, Section 4.1 now explains that Spectrum-full uses the selected singular values to choose the factor scale. Section 4.2 now explains that Forward-moment uses measured activations instead of a Gaussian input model. Its rule requires mutually orthogonal nonzero columns of B. This condition is now beside the rule; its derivation remains in Appendix N.
Section 4.3 now introduces Direct with a short explanation. Measure the output-update magnitudes DA and DB from one temporary optimizer step, restore the temporary changes, and choose c=DB/DA. Then apply B←cB, A←c−1A. This preserves the initial adapter while changing the two factors' update contributions. Ordinary AdamW training starts with a fresh optimizer.
Section 4 now explains the method without requiring an appendix search. It also clarifies that SignSGD is the analysis model, while language-model training uses AdamW.
Q2: "It is hard to track what are the proposals and the reasoning behind them."R2: We derive what ratio to use and compute it automatically. The new result is that different selected singular values need different scales, even at the same width and rank. The Gaussian checks fix input/output sizes at 48/96 and rank at 8. Predicted/observed scales are 0.7708/0.7715 for a flat spectrum. They are 0.5789/0.5799 for a concentrated spectrum. Across 60 configurations, median/maximum prediction errors are 0.11%/0.51%.
The analysis separates the ratio of the two update contributions from their combined size. It also gives the condition that keeps their interaction bounded. This motivates the control in Table 1 that matches the initial combined update size. It scores 33.75/45.30 on PiSSA/MiLoRA, compared with Warm's 34.49/45.76. Choosing the factor ratio brings a benefit beyond matching the initial update size. Section 3.3 now connects this theoretical distinction directly to the matched-size control in Section 5.2.
Q3: "Could the authors compare with LoRA-RITE and LoRA-Muon, or explainable why they are not comparable?"R3: Thank you for suggesting these comparisons. Yes. Warm with AdamW gives 34.49/45.76, close to LoRA-Muon's 34.41/45.86 and LoRA-RITE's 34.37/45.74. We find this comparison useful. Users can reach competitive task scores with a small adjustment before training and keep their familiar AdamW setup.
We compare PiSSA/Qwen3-0.6B and MiLoRA/Qwen3-1.7B under matched training setups. Each optimizer keeps its own update rule. Mean validation negative log-likelihood (NLL) selects one of five learning rates. The table reports its five-seed task means in percent.
A one-time scale adjustment brings AdamW close to these specialized optimizers in the two tested settings.
Revision locations We have revised Sections 4.1–4.3 and added optimizer comparisons in Section 5.4, Table 4.
Please see Part II for the remaining responses.