acceptodds
Under review as a conference paper at ICLR 2027

Normalized Low-Rank Adaptation

Abstract

While low-rank adaptation (LoRA) is widely used for parameter-efficient model adaptation, how to regularize its training dynamics for stable and effective optimization remains underexplored. Because LoRA initializes the up-projection to zero, its early optimization dynamics are largely governed by the down-projection. Building on this observation, we introduce Normalized Low-Rank Adaptation (NoRA), a simple yet effective method that normalizes the down-projection matrices during training. We further show that the same normalization can be applied only at initialization, improving standard LoRA without requiring repeated normalization throughout training. Across pretraining, supervised finetuning, and reinforcement learning, NoRA consistently accelerates convergence, improves performance and training stability, and mitigates catastrophic forgetting. These benefits require neither additional trainable parameters nor inference-time computation, making NoRA a simple and broadly applicable enhancement to LoRA.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM, PEFT, LoRA

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_QEESOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

NoRA normalizes each column of LoRA's down-projection, either throughout training or only at initialization. It preserves a linear, mergeable update. The paper connects this change to early optimization and evaluates pretraining, supervised fine-tuning, and RLVR.

Critical strengths
  1. The change is simple and data-independent. The initialization-only variant is particularly easy to adopt and retains much of the reported benefit without adding an operation during training. This makes the proposal useful without complicating deployment.

  2. SFT results improve over several LoRA variants. At comparable parameter counts, Table 5 reports 43.37, versus 37.93 for LoRA, 41.28 for rsLoRA, and 42.70 for MiSS. The DoRA-based results also improve, supporting the usefulness of the change beyond one parameterization.

Critical weaknesses
  1. Normalization is not separated from overall scale. The main unresolved question is whether equalizing individual column norms helps beyond changing the overall scale of A. The row-normalized control in Table 3 changes that scale too. A uniformly rescaled version of the same random A, matched in Frobenius norm and tuning budget, would test whether per-column equalization adds value beyond initial scale. Equation (16) explicitly includes column correlations, but the experiments do not establish the stronger claim that the diagonal is the decisive quantity. Its link to the AdamW results also needs clarification: a larger reported gradient norm in Figure 1 does not by itself show a better-conditioned weight update.

  2. The stability claim is supported by a limited baseline comparison. Table 4 demonstrates a large recovery from a collapsed baseline under the reported configuration, but does not establish whether a modest change in baseline scale or learning rate would also avoid collapse. This limits how broadly that result can support the stability claim. The pretraining corpus also needs clarification: Tables 2 and 4 name SlimPajama, whereas Appendix A names FineWeb-10BT.

Questions for authors
  1. Does column normalization beat uniform rescaling of the same initial A under matched Frobenius norm and tuning budget? Please identify the gradient quantity in Figure 1 and explain its relationship to the induced update under AdamW.

  2. Is the Table 4 baseline collapse robust to a small scale/learning-rate sweep, and which pretraining corpus was used? Please also clarify Table 6's aggregation: the simple means of its six task averages are 41.62 for NoRA and 40.13 for LoRA, versus the reported 44.4 and 42.8.

3 comments

Response to W1 & Q1

We thank the reviewer for these insightful comments. Our additional scale-matched experiments reveal that most of NoRA's improvements arise from the initialization scale of (A), rather than column-wise equalization alone.

Method(|A|_F)MATHGSM8K
LoRA3.2611.2848.75
NoRA-Col (matched)3.2611.3450.31
NoRA-Row (matched)3.2611.2450.42
NoRA-init55.4014.2459.89
NoRA55.40 (initial)14.4661.63

Unlike manually tuning the scale, NoRA provides a deterministic, dimension-aware calibration rule, yielding ∥A∥F=din\lVert A\rVert_F = \sqrt{d_{\mathrm{in}}} without introducing additional hyperparameters. This is related to LoRA+, which addresses optimization imbalance through asymmetric learning rates, whereas NoRA directly controls AA's scale. Furthermore, NoRA's initial implicit preconditioner P=α2A⊤AP = \alpha^2 A^\top A satisfies Diag⁡(P)=I\operatorname{Diag}(P) = I under unit effective scaling, and E[P]=I\mathbb{E}[P] = I under isotropic initialization, theoretically motivating its scale calibration. Persistent normalization additionally improves performance over NoRA-init.

We agree that diagonal equalization does not guarantee improved conditioning, as off-diagonal correlations remain. Moreover, Figure 1 reflects initial gradient-scale behavior, not necessarily the effective weight update under AdamW, whose adaptive moments modify the update. We will clarify this distinction and revise the theoretical interpretation accordingly. We also acknowledge that a uniformly rescaled version of the same initial (A), with matched Frobenius norm and tuning budget, remains the most direct control for isolating the effect of per-column equalization.

Response to W2.

The learning rate of 3e-4 is the optimal pretraining learning rate recommended by the Flash Linear Attention (FLA) repository. To ensure a fair comparison and isolate the differences between our method and the baseline, we adopted this setting as the default configuration for both methods.

Regarding the PPL collapse, we acknowledge that this phenomenon can occasionally occur at the 340M model scale. To ensure a fair and reliable comparison, we therefore reran the experiments multiple times and carefully verified the results.

ModelScale ParametersLamb.†Wiki. Lamb.ARCeARCcHella.PIQAWino.OBQAAvg.
ppl↓ppl↓acc↑acc↑accn↑accn↑acc↑acc↑acc↑acc↑
MHA with 10B training tokens and 0.5M batchsize tokens
W x348.7M47.9531.3130.5554.5926.4536.9565.0751.2222.6041.06
BAx289.3M74.3140.2325.6753.9226.0133.8663.5250.7918.4638.89
BANoRA-initx289.3M63.4534.3928.2954.3826.0235.1063.4950.9119.6039.69

We apologize for the inconsistency. The actual pretraining dataset was FineWeb-Edu; the references to SlimPajama in Tables 2 and 4 were writing errors and have been corrected. Our experiments were implemented using the publicly available FLA framework, facilitating reproducibility.

Response to Q2

We have rerun the experiments reported in Table 4. RL

MethodAIME24@32 Avg.AIME24@32 PassAIME25@32 Avg.AIME25@32 PassAMC@32 Avg.AMC@32 PassHMMT@32 Avg.HMMT@32 PassMATH500@4 Avg.MATH500@4 PassMinerva@4 Avg.Minerva@4 PassAvg.
Base24.3070.0020.4053.3064.8095.009.9036.7080.9092.8027.8043.0038.02
MiLoRA18.3360.0017.8140.0061.88100.008.1838.3077.0590.2025.2841.2034.76
PiSSA3.4420.000.8313.3033.9180.001.8216.7051.5076.6015.7230.9017.87
LoRA28.0060.0023.1053.3068.0097.5010.3035.0081.3092.2030.1046.3040.13
NoRA26.7066.7023.5046.7072.7097.5010.3035.0084.5092.8032.0047.8041.62

The incorrect results in Table 6 were caused by a manual calculation error, which we have now corrected. Additionally, we have rerun the PiSSA and MiLoRA experiments to verify the results and ensure consistency.

Thank you for the additional experiments and corrections. The corpus clarification and corrected Table 6 resolve the reporting issues. However, the response confirms that most of the gain comes from the initialization scale of A rather than column-wise equalization. The key control is still missing: uniformly rescaling the same random A to the NoRA-init norm of 55.40 under the same tuning budget. The NoRA-init versus NoRA difference is also reported without uncertainty, and the stability claim is not tested with a learning-rate or scale sweep. These results do not resolve my main concern, so I will keep my rating unchanged.

We thank the reviewer for the continued feedback and constructive suggestions.

First, we would like to clarify that the norm of 55.40 is not a manually tuned scaling factor, but is directly determined by NoRA's dimension-aware column normalization. This deterministic scale calibration, without introducing additional hyperparameters, is itself part of NoRA's contribution.

To further examine robustness across learning rates, we conducted additional experiments using the official Hugging Face PEFT framework.

GSM8K Test Accuracy (%)

MethodLR=1e-4LR=5e-5LR=2e-5LR=1e-5
LoRA46.8543.6739.9238.36
NoRA-init48.8248.0743.2140.86
DoRA45.7244.4340.1438.44
PiSSA47.5446.1042.7239.50

Under the same training configuration (batch size = 4, 5,000 steps, approximately 20K MetaMathQA samples), NoRA-init consistently outperforms LoRA, DoRA, and PiSSA across all four learning rates.

These results demonstrate that NoRA's initialization provides benefits across a range of learning rates, rather than at a single carefully selected learning rate. We acknowledge that a uniformly rescaled random initialization matched to the same Frobenius norm would help further isolate the contribution of column-wise equalization. Nevertheless, we emphasize that NoRA's contribution also includes a deterministic, dimension-aware scale calibration rule, rather than merely the equalization of column norms.

We appreciate the reviewer's suggestions and will clarify this distinction in the revised manuscript.

Reviewer_ENM4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

To address the inefficient early-stage optimization and training instability associated with imbalanced projection scales across input dimensions in standard LoRA, this paper proposes NoRA, which normalizes the columns of the down-projection matrix across the rank dimension to control the scale of the input-to-latent projection, and further introduces NoRA-init, which applies normalization only at initialization. Experiments across pretraining, supervised fine-tuning, and reinforcement learning demonstrate the effectiveness of the proposed methods.

Critical strengths
  1. The proposed NoRA method is simple and computationally efficient, requiring no additional trainable parameters or inference-time overhead while preserving the weight mergeability of LoRA.
  2. The introduction of NoRA-init is practically valuable, as initialization-only normalization captures a substantial portion of the performance gains of NoRA without requiring persistent normalization during training.
Critical weaknesses
  1. The novelty of NoRA appears limited compared with existing work on LoRA initialization, scaling, and optimization. Its main contribution is column-wise normalization of the down-projection matrix, but the paper does not sufficiently establish a fundamentally new optimization principle or demonstrate how this mechanism addresses limitations that existing approaches cannot adequately resolve.
  2. The comparison between row-wise and column-wise normalization is confounded by differences in the overall scale of the resulting matrices. Without scale-matched controls, it remains unclear whether the performance gains achieved by NoRA arise from column-wise equalization itself or from changes in the scale of the initialized matrices.
  3. Equalizing the diagonal of the implicit preconditioner does not guarantee better conditioning, as correlations between different coordinates remain unconstrained. Moreover, matching adapter gradient norms to those of full fine-tuning does not establish comparable weight updates, leaving the claimed optimization benefits insufficiently justified.
  4. The paper provides inconsistent descriptions of the pretraining dataset: Table 2 and Table 4 identify SlimPajama, whereas Appendix A specifies FineWeb-10BT. This discrepancy leaves the actual data used in the pretraining experiments unclear and undermines reproducibility.
Questions for authors
  1. Does exact column-wise normalization outperform a globally rescaled random initialization with the same expected overall scale, or a row-normalized initialization rescaled to match the Frobenius norm of the column-normalized matrix?
  2. Which initialization scheme is used for NoRA-init in Table 5, and can the authors compare NoRA and NoRA-init using the same initial factor matrices and matched overall scale to isolate the contribution of persistent normalization?
  3. What evidence demonstrates that equalizing the diagonal entries of the implicit preconditioner is the key mechanism underlying the performance gains of NoRA, rather than changes in overall scale or uncontrolled cross-coordinate correlations?
3 comments

Response to W1:

We thank the reviewer for this comment. As discussed in Sections 3.1–3.3 and Table 1, NoRA differs from existing methods by directly controlling the scale of the down-projection matrix while preserving LoRA's linear structure and exact mergeability.

Our additional experiments reveal that initialization scale accounts for most of NoRA's performance improvements. Importantly, NoRA provides a principled, dimension-aware scale calibration rule, yielding (|A|F=\sqrt{d{\mathrm{in}}}) without introducing an additional scale hyperparameter. As analyzed in Section 3.3, this rule calibrates the initial implicit preconditioner and aligns the expected initial SGD update with full fine-tuning under the stated assumptions.

Unlike LoRA+, which explicitly adjusts factor-wise learning rates, NoRA directly controls the initialization scale and optionally maintains normalization throughout training. Unlike PiSSA and MiLoRA, it requires neither SVD nor modifications to the pretrained weights.

We acknowledge that NoRA does not introduce a fundamentally new optimizer. Rather, its contribution lies in identifying and validating a simple, theoretically motivated scale calibration mechanism for low-rank adaptation, supported by experiments across pretraining, SFT, and RLVR, without additional trainable parameters or inference-time overhead.

Response to W2/Q1.

We thank the reviewer for these insightful comments. Our additional scale-matched experiments reveal that most of NoRA's improvements arise from the initialization scale of (A), rather than column-wise equalization alone.

Method(|A|_F)MATHGSM8K
LoRA3.2611.2848.75
NoRA-Col (matched)3.2611.3450.31
NoRA-Row (matched)3.2611.2450.42
NoRA-init55.4014.2459.89
NoRA55.40 (initial)14.4661.63

Unlike manually tuning the scale, NoRA provides a deterministic, dimension-aware calibration rule, yielding ∥A∥F=din\lVert A\rVert_F = \sqrt{d_{\mathrm{in}}} without introducing additional hyperparameters. This is related to LoRA+, which addresses optimization imbalance through asymmetric learning rates, whereas NoRA directly controls AA's scale. Furthermore, NoRA's initial implicit preconditioner P=α2A⊤AP = \alpha^2 A^\top A satisfies Diag⁡(P)=I\operatorname{Diag}(P) = I under unit effective scaling, and E[P]=I\mathbb{E}[P] = I under isotropic initialization, theoretically motivating its scale calibration. Persistent normalization additionally improves performance over NoRA-init.

We agree that diagonal equalization does not guarantee improved conditioning, as off-diagonal correlations remain. Moreover, Figure 1 reflects initial gradient-scale behavior, not necessarily the effective weight update under AdamW, whose adaptive moments modify the update. We will clarify this distinction and revise the theoretical interpretation accordingly. We also acknowledge that a uniformly rescaled version of the same initial (A), with matched Frobenius norm and tuning budget, remains the most direct control for isolating the effect of per-column equalization.

Response to Q3/W3

We thank the reviewer for this important observation. We agree that equalizing the diagonal entries of the implicit preconditioner does not guarantee improved conditioning, as off-diagonal correlations remain unconstrained.

Our theoretical analysis establishes a more specific property: NoRA deterministically enforces (Diag⁡(P)=I\operatorname{Diag}(P)=I) under unit effective scaling, while isotropic initialization yields (E[P]=I\mathbb{E}[P]=I). This provides a principled calibration of the initial optimization scale rather than a guarantee of improved conditioning.

We also acknowledge that matching initial gradient norms does not imply equivalent effective weight updates under AdamW. The analysis in Section 3.3 characterizes the initial SGD update, whereas AdamW introduces adaptive moment-based scaling. Therefore, Figure 1 should be interpreted as evidence of initial gradient-scale alignment, not update equivalence.

Importantly, our additional scale-matched experiments reveal that initialization-scale calibration, rather than diagonal equalization alone, explains most of NoRA's empirical improvements. We will revise the manuscript to clarify these distinctions and avoid overstating the theoretical implications.

Response to W4.

We apologize for the inconsistency. The actual pretraining dataset was FineWeb-Edu; the references to SlimPajama in Tables 2 and 4 were writing errors and have been corrected. Our experiments were implemented using the publicly available FLA framework, facilitating reproducibility.

Response to Q2

Thank you for the question. NoRA-init in Table 5 uses the default Kaiming initialization, followed by column-wise normalization. Unlike NoRA, which maintains normalization throughout training, NoRA-init applies it only at initialization.

As demonstrated in Table 3, initialization alone is not the primary factor driving NoRA's improvements. The superior performance of NoRA over NoRA-init in Table 5 further suggests that persistent normalization provides additional benefits beyond initialization.

Thank you to the authors for their efforts in addressing my concerns. However, I believe the distinct contribution of NoRA remains insufficiently established. The scale-matched results show little advantage of column-wise over row-wise normalization, while the missing global-rescaling control leaves unclear whether dimension-aware calibration offers benefits beyond simpler scaling strategies. Consequently, the novelty and independent value of the proposed calibration mechanism remain unclear. Therefore, I maintain my original rating of 2.

Reviewer_xRdLOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes NoRA, which normalizes the down-projection matrix of LoRA column-wise along the rank dimension. It starts from the observation that LoRA initializes the up-projection to zero, so the early optimization is determined by the down-projection, and it applies the latent normalization effect of MLA to the parameters rather than the input, which keeps linearity and mergeability. It presents NoRA, which applies the normalization throughout training, and NoRA-init, which applies it only once at initialization. Theoretically, it views the initial LoRA update as the full fine-tuning gradient multiplied by a preconditioner determined by the down-projection, and explains that since the column norms act as per-input-coordinate learning rates, the normalization sets them to 1 and matches the gradient magnitude to that of full fine-tuning.

Critical strengths

(S1) The method is simple and clear. It only requires normalizing A in standard LoRA, so the implementation is also very simple.

(S2) The paper is well written and easy to follow.

Critical weaknesses

(W1) The comparison with prior work is weak.

DoRA learns direction and magnitude separately, whereas NoRA fixes the magnitude of A to 1 and lets only B determine the magnitude. The difference from this existing method should be addressed in a discussion.

NoRA also appears comparable to LoRA+. LoRA+ argues that because A and B have asymmetric shapes, B is not trained enough under the same learning rate, and it sets the learning rate of B higher than that of A. The normalization in NoRA ends up making A considerably larger than the default initialization. Under Adam, this has an effect similar to raising the learning rate of B and lowering that of A, so NoRA-init appears to overlap substantially with implementing the LoRA+ learning rate ratio through initialization. If so, a large part of the gains reported in the paper (initial gradient norm, convergence speed, SFT performance) could also be explained by the LoRA+ mechanism. LoRA+ appears in Table 1 but is missing from the experimental comparison. Adding LoRA+ with a tuned learning rate and an initialization that only scales up A without normalization as control groups would make it much clearer how much the normalization itself contributes. In particular, validating NoRA in the experimental setting of LoRA+ is recommended.

(W2) Were all experiments run by the authors themselves? When fine-tuning with LoRA, experiments over multiple seeds are necessary. Also, the excessive performance drop in Table 6 is questionable. Does the same sharp performance drop appear with other seeds or other hyperparameters?

Related to this concern, it should be pointed out that the authors do not demonstrate performance in the experimental settings conventionally used in LoRA-family papers. For example, they did not run the commonly used evaluation settings such as fine-tuning BERT-family models on GLUE as in LoRA original paper, or training LLaMA-7B/8B on Commonsense Reasoning-170k with LoRA. In addition, since the experimental scale is limited to 340M to 3B, there is a concern about scale. Therefore, the authors need to compare the official numbers reported in other papers with the results they obtain themselves in those frameworks.

The implementation looks easy, but since the rebuttal period is limited, running some GLUE tasks on BERT-family models to show performance is recommended. (However, also testing datasets of some size, such as SST-2, would help demonstrate performance.)

(W3) The asymmetry between A and B has been studied extensively. In particular, [1] reports that an untrained A achieves the same performance as LoRA. A qualitative and quantitative comparison with this is needed.

(W4) It is unclear whether the reported gains are distinguishable from the change in effective learning rate caused by the larger scale of A. Please add LoRA+ to the comparison with a ratio sweep and learning rate tuning. Also, please run an initialization that only matches the scale of A to NoRA without normalization, and compare it with NoRA-init under the same settings.

Asymmetry in Low-Rank Adapters of Foundation Models, ICML24

Questions for authors

See weaknesses.

6 comments

Response to W1.

We thank the reviewer for highlighting the connection between NoRA and LoRA+. We clarify that, unlike DoRA, which explicitly decomposes pretrained weights into magnitude and direction, NoRA directly normalizes the columns of the LoRA down-projection matrix without introducing additional magnitude parameters.

Although NoRA and LoRA+ can both influence the optimization dynamics of A and B, their mechanisms differ: LoRA+ explicitly employs asymmetric learning rates, whereas NoRA modifies the column-wise scale of A through normalization. Moreover, NoRA maintains this normalization throughout training, beyond initialization-time rescaling.

To address the reviewer's concern, we have added the standard LoRA+ baseline to our SFT experiments and evaluated it at both 3B and 8B model scales. These additional comparisons provide a direct evaluation of NoRA against LoRA+ under the same experimental setting.

Supplementary Table 5:

MethodTrainableGSM8KMathHumanEvalMBPPAvgMMLUAGIEvalARC-CAvg Δ
Base Model-----------55.10 (+0.00)23.99 (+0.00)42.92 (+0.00)0.00
Full3B65.1217.9636.1549.0542.0754.17 (-0.93)26.25 (+2.26)41.64 (-1.28)+0.02
LoRA48.6M50.9410.3837.2053.2037.9354.27 (-0.83)24.40 (+0.42)41.64 (-1.28)-0.56
LoRA+48.6M58.3213.7438.4055.0041.3754.32 (-0.78)24.13 (+0.14)41.55 (−1.37)-0.61
NoRA-init48.6M59.8914.2439.6055.8042.3854.09 (-1.01)25.10 (+1.11)42.24 (-0.68)-0.19

Llama-3.1-8B

MethodTrainableGSM8KMathHumanEvalMBPPAvgMMLUAGIEvalARC-CAvg Δ
Base Model-----------64.23 (+0.00)35.53 (+0.00)52.39 (+0.00)0.00
LoRA83.9M72.5524.4647.6064.6052.3063.52 (-0.71)35.73 (+0.19)52.05 (-0.34)-0.29
PiSSA83.9M75.7426.2050.0068.0054.9862.80 (-1.43)36.20 (+0.67)52.82 (+0.43)-0.11
LoRA+83.9M76.3527.6052.4068.0056.0963.00 (-1.23)35.80 (+0.27)51.79 (-0.60)-0.52
OFT79.1M73.5426.3048.8067.2053.9664.07 (-0.16)36.04 (+0.51)53.07 (+0.68)+0.34
DoRA85.3M72.2524.6648.2066.9053.0063.45 (-0.78)36.37 (+0.83)51.96 (-0.43)-0.12
MiSS88.1M77.2628.1855.5068.8057.4362.33 (-1.90)37.83 (+2.30)51.96 (-0.43)-0.01
NoRA83.9M76.2728.6656.7069.6057.8162.73 (-1.50)36.95 (+1.42)52.22 (-0.17)-0.09

Response to W2.

All experiments in this paper were conducted by the authors, with multiple random seeds used throughout our evaluations. The performance collapse observed with PiSSA and MiLoRA occurs inconsistently across runs, and we suspect that it may be related to the rollout process. In the original submission, we used the corresponding results from PEFT. To ensure a fair comparison, we have rerun these experiments using multiple random seeds and updated the reported results accordingly.

Additional Evaluation on BERT-base

We further evaluate NoRA on BERT-base-cased across all nine GLUE tasks. Both methods use rank 8, a learning rate of 5×10−45\times10^{-4}, and three training epochs, with identical AdamW optimization and other training settings. Results are averaged over 3 random seeds (0 ,7,42) on the development sets. NoRA achieves better or comparable performance on CoLA, MRPC, STS-B, and WNLI, while showing modest declines on QQP, MNLI, and QNLI. Overall, NoRA achieves comparable average performance to LoRA (79.85 vs. 79.97 across eight tasks, excluding WNLI), providing additional evidence of its effectiveness beyond the previously evaluated model architectures.

TaskMetricLoRA (r8/a16)NoRA (r8/a8)Δ (pp)
CoLAMatthews51.1453.00+1.86
SST-2Accuracy90.4290.31-0.11
MRPCAccuracy84.0785.29+1.23
STS-BPearson87.9588.03+0.08
QQPAccuracy88.2987.28-1.01
MNLIAccuracy (matched)82.2080.33-1.87
QNLIAccuracy90.3389.23-1.11
RTEAccuracy65.3465.340.00
WNLIAccuracy48.5952.82+4.23
Average (8 tasks, excluding WNLI)—79.9779.85-0.12

Experimental details: We use BERT-base-cased (108.6M parameters), a maximum sequence length of 128, and a batch size of 32 per GPU. LoRA uses standard Kaiming-uniform initialization for AA and zero initialization for BB, while NoRA normalizes each column of AA to unit norm, also with zero-initialized BB. Both use zero dropout, AdamW (β1=0.9\beta_1=0.9, β2=0.999\beta_2=0.999, ϵ=10−8\epsilon=10^{-8}), zero weight decay, linear learning-rate decay without warmup, and gradient clipping at 1.0.

RL

MethodAIME24@32 Avg.AIME24@32 PassAIME25@32 Avg.AIME25@32 PassAMC@32 Avg.AMC@32 PassHMMT@32 Avg.HMMT@32 PassMATH500@4 Avg.MATH500@4 PassMinerva@4 Avg.Minerva@4 PassAvg.
Base24.3070.0020.4053.3064.8095.009.9036.7080.9092.8027.8043.0038.02
MiLoRA18.3360.0017.8140.0061.88100.008.1838.3077.0590.2025.2841.2034.76
PiSSA3.4420.000.8313.3033.9180.001.8216.7051.5076.6015.7230.9017.87
LoRA28.0060.0023.1053.3068.0097.5010.3035.0081.3092.2030.1046.3040.13
NoRA26.7066.7023.5046.7072.7097.5010.3035.0084.5092.8032.0047.8041.62

Note: The results above were obtained using two random seeds, and the experimental setting follows Perl alignment.

Response to W3/4.

We thank the reviewer for pointing out this related work. Our analysis shows that MiSS can be viewed as a variant of LoRA in which A is fixed and only B is trained, providing a relevant comparison to methods using an untrained A. Furthermore, we have added LoRA+ to our SFT experiments to compare against an approach that explicitly addresses the optimization asymmetry between A and B. Together, these analyses and experiments provide additional evidence for understanding the role of A and its optimization in NoRA.

Thank you for the detailed rebuttal and the additional experiments. However, my concerns still remain.

First, since the code is not available, it is difficult to verify the implementation.

PEFT layers are generally simple to implement, so I asked the authors to follow, within the rebuttal period, the experimental frameworks used in related work such as LoRA, LoRA+, and DoRA. However, the authors again conducted the experiments in their own setting. In addition, the BERT experiment in the response to W2 was trained for only 3 epochs, so the models appear to be under-trained. In terms of performance, there is either no improvement (BERT GLUE experiment) or the improvements are mostly marginal. Since most of my concerns were about the comparison with related work and the experimental setting, and these concerns have not been resolved, I will maintain my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.