- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
NoRA normalizes each column of LoRA's down-projection, either throughout training or only at initialization. It preserves a linear, mergeable update. The paper connects this change to early optimization and evaluates pretraining, supervised fine-tuning, and RLVR.
Critical strengths
-
The change is simple and data-independent. The initialization-only variant is particularly easy to adopt and retains much of the reported benefit without adding an operation during training. This makes the proposal useful without complicating deployment.
-
SFT results improve over several LoRA variants. At comparable parameter counts, Table 5 reports 43.37, versus 37.93 for LoRA, 41.28 for rsLoRA, and 42.70 for MiSS. The DoRA-based results also improve, supporting the usefulness of the change beyond one parameterization.
Critical weaknesses
-
Normalization is not separated from overall scale. The main unresolved question is whether equalizing individual column norms helps beyond changing the overall scale of A. The row-normalized control in Table 3 changes that scale too. A uniformly rescaled version of the same random A, matched in Frobenius norm and tuning budget, would test whether per-column equalization adds value beyond initial scale. Equation (16) explicitly includes column correlations, but the experiments do not establish the stronger claim that the diagonal is the decisive quantity. Its link to the AdamW results also needs clarification: a larger reported gradient norm in Figure 1 does not by itself show a better-conditioned weight update.
-
The stability claim is supported by a limited baseline comparison. Table 4 demonstrates a large recovery from a collapsed baseline under the reported configuration, but does not establish whether a modest change in baseline scale or learning rate would also avoid collapse. This limits how broadly that result can support the stability claim. The pretraining corpus also needs clarification: Tables 2 and 4 name SlimPajama, whereas Appendix A names FineWeb-10BT.
Questions for authors
-
Does column normalization beat uniform rescaling of the same initial A under matched Frobenius norm and tuning budget? Please identify the gradient quantity in Figure 1 and explain its relationship to the induced update under AdamW.
-
Is the Table 4 baseline collapse robust to a small scale/learning-rate sweep, and which pretraining corpus was used? Please also clarify Table 6's aggregation: the simple means of its six task averages are 41.62 for NoRA and 40.13 for LoRA, versus the reported 44.4 and 42.8.
3 comments
Thank you for the additional experiments and corrections. The corpus clarification and corrected Table 6 resolve the reporting issues. However, the response confirms that most of the gain comes from the initialization scale of A rather than column-wise equalization. The key control is still missing: uniformly rescaling the same random A to the NoRA-init norm of 55.40 under the same tuning budget. The NoRA-init versus NoRA difference is also reported without uncertainty, and the stability claim is not tested with a learning-rate or scale sweep. These results do not resolve my main concern, so I will keep my rating unchanged.
We thank the reviewer for the continued feedback and constructive suggestions.
First, we would like to clarify that the norm of 55.40 is not a manually tuned scaling factor, but is directly determined by NoRA's dimension-aware column normalization. This deterministic scale calibration, without introducing additional hyperparameters, is itself part of NoRA's contribution.
To further examine robustness across learning rates, we conducted additional experiments using the official Hugging Face PEFT framework.
GSM8K Test Accuracy (%)
| Method | LR=1e-4 | LR=5e-5 | LR=2e-5 | LR=1e-5 |
|---|---|---|---|---|
| LoRA | 46.85 | 43.67 | 39.92 | 38.36 |
| NoRA-init | 48.82 | 48.07 | 43.21 | 40.86 |
| DoRA | 45.72 | 44.43 | 40.14 | 38.44 |
| PiSSA | 47.54 | 46.10 | 42.72 | 39.50 |
Under the same training configuration (batch size = 4, 5,000 steps, approximately 20K MetaMathQA samples), NoRA-init consistently outperforms LoRA, DoRA, and PiSSA across all four learning rates.
These results demonstrate that NoRA's initialization provides benefits across a range of learning rates, rather than at a single carefully selected learning rate. We acknowledge that a uniformly rescaled random initialization matched to the same Frobenius norm would help further isolate the contribution of column-wise equalization. Nevertheless, we emphasize that NoRA's contribution also includes a deterministic, dimension-aware scale calibration rule, rather than merely the equalization of column norms.
We appreciate the reviewer's suggestions and will clarify this distinction in the revised manuscript.
Response to W1 & Q1
We thank the reviewer for these insightful comments. Our additional scale-matched experiments reveal that most of NoRA's improvements arise from the initialization scale of (A), rather than column-wise equalization alone.
Unlike manually tuning the scale, NoRA provides a deterministic, dimension-aware calibration rule, yielding ∥A∥F=din without introducing additional hyperparameters. This is related to LoRA+, which addresses optimization imbalance through asymmetric learning rates, whereas NoRA directly controls A's scale. Furthermore, NoRA's initial implicit preconditioner P=α2A⊤A satisfies Diag(P)=I under unit effective scaling, and E[P]=I under isotropic initialization, theoretically motivating its scale calibration. Persistent normalization additionally improves performance over NoRA-init.
We agree that diagonal equalization does not guarantee improved conditioning, as off-diagonal correlations remain. Moreover, Figure 1 reflects initial gradient-scale behavior, not necessarily the effective weight update under AdamW, whose adaptive moments modify the update. We will clarify this distinction and revise the theoretical interpretation accordingly. We also acknowledge that a uniformly rescaled version of the same initial (A), with matched Frobenius norm and tuning budget, remains the most direct control for isolating the effect of per-column equalization.
Response to W2.
The learning rate of 3e-4 is the optimal pretraining learning rate recommended by the Flash Linear Attention (FLA) repository. To ensure a fair comparison and isolate the differences between our method and the baseline, we adopted this setting as the default configuration for both methods.
Regarding the PPL collapse, we acknowledge that this phenomenon can occasionally occur at the 340M model scale. To ensure a fair and reliable comparison, we therefore reran the experiments multiple times and carefully verified the results.
We apologize for the inconsistency. The actual pretraining dataset was FineWeb-Edu; the references to SlimPajama in Tables 2 and 4 were writing errors and have been corrected. Our experiments were implemented using the publicly available FLA framework, facilitating reproducibility.
Response to Q2
We have rerun the experiments reported in Table 4. RL
The incorrect results in Table 6 were caused by a manual calculation error, which we have now corrected. Additionally, we have rerun the PiSSA and MiLoRA experiments to verify the results and ensure consistency.