acceptodds
Under review as a conference paper at ICLR 2027

Learn the Directions, Normalize the Gains: Post-Training Normalization for LoRA

Abstract

While Low-Rank Adaptation (LoRA) enables efficient task specialization, its learned updates can compromise capabilities beyond the target task. We identify **adaptation imbalance**: a few singular directions dominate the trained update, leaving its performance sensitive to how gains are allocated. We argue that **learning where to adapt does not ensure that adaptation gains are well balanced**. This motivates **LoRA-Norm**, a post-training normalization method that retains learned directions while rebalancing their gains. LoRA-Norm combines spectral rebalancing, a fixed nonlinear transformation of singular values, with nuclear-norm restoration, which preserves the original total spectral mass. It requires no calibration data or additional training and introduces no inference overhead. Across two backbones and three adaptation tasks, LoRA-Norm improves average specialization and capability retention, outperforming the evaluated post-hoc spectral pruning and gradient-guided editing configurations on both measures. Stronger functional equalization brings no consistent additional gains, revealing that balancing adapter gains and equalizing their responses are distinct objectives.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Low-Rank Adaptation (LoRA), Parameter-Efficient Fine-Tuning (PEFT), Training-Free Refinement, Large Language Models

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_XJ3fOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes LoRA Norm, a post-training method for editing completed LoRA adapters without calibration data or additional training. It preserves the updates learned singular directions, transforms its singular values using a fixed hybrid Newton–Schulz procedure, and restores each module’s original nuclear norm.

The edited adapter retains the original rank and inference structure.

Evaluation covers 18 rank-16 adapters spanning two 8B backbones, three adaptation tasks, and three training runs per model–task pair. Across these checkpoints, average target performance increases from 68.00% to 72.08%, off-task performance increases from 69.55% to 72.79%, and the forgetting gap decreases from 2.71 to 0.34 percentage points. Comparisons include spectral pruning, gradient-guided editing, and several normalization controls.

The analysis shows that normalization also substantially reduces functional update energy. Energy-matched scalar controls reproduce much of the benefit, while stronger functional equalization provides no consistent additional improvement. Consequently, the evidence supports a useful post-training normalization procedure, with limited evidence for an independent advantage from spectral balancing itself.

I like the approach and the simplicity of the idea, and the fact that it does not add overhead.

Critical strengths
  1. The authors present a practical intervention with consistent (average) benefits. The method uses only adapter parameters, requires no further optimization, and preserves the inference structure.

b. Target performance improves on the majority of checkpoints, and off-task performance also usually improves. Simultaneous average improvements in specialization and retention make this a potentially useful deployment procedure.

c. The analysis is useful and helps clarify the results --- Scalar-E recovers much of the improvement without changing each module's participation ratio, while stronger functional equalization does not consistently improve performance.
This suggests that reducing the update strength contributes to the gains, and that making the directional contributions more uniform is not necessarily better.

In short, the paper presents a simple method with no inference overhead, demonstrates practical improvements, and provides a nice analysis. I therefore lean toward acceptance.

Critical weaknesses
  1. The advantage of the specific hybrid transformation over simpler alternatives is still unclear to me. Flat-Nuclear is also data-free and is only 0.15 percentage points behind LoRA-Norm on target performance (Table 4), but its off-task and forgetting results are not reported in that comparison (which is very importnant). Scalar-E also achieves a higher average target score, although the difference is not statistically significant, it requires activation data, and LoRA-Norm has a lower forgetting gap (Tables 17 and 22). The authors acknowledge these limitations. Still, while the results support adjusting gains after training, it is less clear when this particular transformation is preferable.

  2. Reconstruction itself changes some results, but this is not fully controlled in the main evaluation. Appendixes shows that reconstructing the adapter without changing its singular values improves target performance by about 0.47 percentage points on average across six archived checkpoints, and that different factorizations can change predictions. However, the same reconstruction-only control is not reported for the main 18 checkpoints. This does not invalidate the overall gains, but it makes it harder to separate small improvements from numerical effects introduced by reconstruction.

Questions for authors
  1. Can you evaluate Flat-Nuclear using the same setup as the main results and report Target, Off, and FG with paired confidence intervals? This would help clarify whether HNS provides a meaningful benefit over exact flattening.

  2. Can you report Target, Off, and FG for reconstruction alone on the main 18 checkpoints, without changing the singular values and using the same reconstruction and inference setup as LoRA-Norm? Do the conclusions remain the same when comparing against this baseline?

  3. Could you also compare with the smooth_abs policy from Spectral Surgery? The original paper describes it as a robust default and reports large performance drops for grad_direction on some tasks. Would including it change the conclusions from Table 2?

Could you also report LoRA-Norm restricted to o_proj and down_proj, using the same checkpoints and evaluation setup? This would help clarify whether its advantage over Spectral Surgery remains when both methods edit the same module families.

Reviewer_FzQ5OpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces LoRA-Norm, a post-training procedure that transforms adapter singular values while preserving their associated directions and per-module nuclear norm. It improves average target-task performance and off-task capability retention without calibration data or retraining.

Critical strengths
  1. The method is simple and inexpensive, preserves the adapter configuration, and adds no inference operations.

  2. The paired controls provide useful evidence about spectral redistribution and update strength.

Critical weaknesses
  1. The additional value of the specific normalization remains insufficiently established. Energy-matched scaling reproduces much of the benefit. These findings leave the necessity of the proposed two-stage transformation unclear.

  2. The baseline comparison does not isolate the effect of the editing rule. Spectral Surgery edits only two module families, whereas LoRA-Norm edits seven. Consequently, the reported advantage cannot be attributed solely to the proposed normalization.

  3. The relationship to related subspace and recalibration approaches could be better positioned. In particular, LoDA [1] combines task-driven shared/isolated subspaces with post-training recalibration. Its setting differs from that of LoRA-Norm, but discussing their distinct assumptions, mechanisms, and potential compatibility would clarify the contribution.

  4. As a minor presentation issue, the complete editing procedure is distributed across the main text and appendix. A consolidated algorithm specifying inputs, outputs, module selection, decomposition, spectral transformations, normalization, and numerical safeguards would improve clarity and reproducibility.

[1] Task-Driven Subspace Decomposition for Knowledge Sharing and Isolation in LoRA-based Continual Learning

Questions for authors
  1. Does LoRA-Norm consistently outperform exact nuclear-norm flattening and simple scaling under the same primary evaluation protocol?

  2. Does its advantage over Spectral Surgery persist when target modules and tuning budgets are matched?

  3. How does the proposed normalization differ from LoDA’s subspace construction and post-training recalibration? Where applicable, could a controlled comparison or combination study clarify their respective benefits?

  4. Are the gains robust across adapter ranks, model sizes, and source checkpoints trained with a consistent, well-tuned recipe?

Reviewer_8EucOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes LoRA-Norm, a training-free post-processing method for improving LoRA adapters. The authors observe that trained LoRA updates often exhibit imbalanced singular value distributions, with a few dominant directions accounting for most of the adaptation strength. To address this, LoRA-Norm applies SVD to the trained LoRA updates, rebalances their singular values using a fixed Newton–Schulz transformation, and restores the original nuclear norm while preserving the learned singular directions. The paper analyses spectral balance and functional update energy, comparing with some simple energy-matched scaling methods.

Critical strengths
  • LoRA-Norm can be directly applied to trained LoRA adapters without additional training or calibration data.
  • The method improves both target-task performance and capability retention across multiple tasks. The ablation studies also provide useful insights into the relationship between spectral rebalancing and functional update energy.
Critical weaknesses
  • LoRA-Norm mainly combines existing techniques, including SVD-based decomposition, Newton–Schulz transformations adopted from Muon, and nuclear-norm restoration. Prior work such as Spectral Surgery has explored a very similar approach of adjusting singular values while keeping singular directions fixed. The authors should better clarify the key novelty of LoRA-Norm and justify whether all these components are necessary. The Scalar-E baseline achieves comparable performance without changing the singular-value distribution. The paper does not provide sufficient evidence that spectral rebalancing offers independent benefits beyond appropriate scaling.
  • Several experiments use separate evaluation protocols, making direct comparisons difficult. A more unified experimental presentation would strengthen the conclusions.
Questions for authors
  • The original LoRA adapters may be underoptimized or overfitted. Do the improvements persist when using better-tuned learning rates, longer training schedules, or validation-selected checkpoints? Does the method remain effective across different LoRA ranks and training strategies?
  • Why does simply modifying the singular values improve both target-task performance and capability retention? Are the gains mainly attributable to reduced update magnitude, or spectral rebalancing? Could the authors provide more controlled experiments to distinguish these explanations?

Comments on the paper

4 comments
Area_Chair_9kacOpenReview ↗

Discussion deadline approaching

Dear Reviewers and Authors,

Thank you for providing detailed reviews and rebuttal.

To the authors: After carefully reading the reviews and rebuttal, I have a clarification question for the authors regarding the paper’s position on the main source of LoRA-Norm’s gains.

The title, abstract, beginning of the introduction, and the reply to Reviewer 8Euc (Q2) present spectral rebalancing as the main contributor. However, the end of the introduction and Section 5 indicate that energy-matched scaling with the original spectrum shape (Scalar-E) recovers most of the benefit.

So what is the main contributor: shape or magnitude? Does Scalar-E correspond to “original shape + LoRA-Norm’s magnitude (in terms of functional energy)”? Why is the Frobenius norm, rather than functional energy, used as the magnitude axis in the analysis in Q2? Would it be fair to say that LoRA-Norm works mainly by reducing per-module magnitude, with rebalancing being a data-free way to find suitable reduction coefficients rather than a necessary ingredient in itself? If so, would you align the title, abstract and introduction with this view?

Kind regards,

AC

Response to AC 9kac

Dear Area Chair,

Thank you for the question. It concerns the point that all three reviewers raised independently (8Euc W1/Q2, FzQ5 W1, XJ3f W1) and that the controls added in the rebuttal were designed to test. Our position is as follows.

Spectral rebalancing is the cause of LoRA-Norm’s gains; reduced functional magnitude is the pathway through which it acts. The two are not competing explanations.

1. “Shape or magnitude” is not a dichotomy for this operation. At fixed nuclear norm, flattening necessarily reduces update strength (Eq. 7; Section 5.1 states that LoRA-Norm “changes both directional allocation and update strength”). The magnitude reduction is a consequence of rebalancing, set per module by that module’s own imbalance. The opening of the paper describes the operation, and Section 5 analyzes how it acts. These are two levels of one account.

2. Scalar-E is “original shape + LoRA-Norm’s functional energy”, and that is why it is not a magnitude-only alternative. Its per-module scale is computed from LoRA-Norm’s output t together with activation data (Eq. 8). Without LoRA-Norm there is no Scalar-E. Its tie on Target and Off shows that rebalancing acts through per-module functional strength. It does not show that this strength can be obtained without rebalancing.

3. Rebalancing is what moves functional magnitude in the data-free setting. We add the functional-energy axis you ask about to the Q2 factorial (same 18 checkpoints and activation statistics as Section 5):

Data-free editFrobenius normEnergy / LoRATarget ↑FG ↓
Recon-0 (no edit)original1.00068.482.58
Scale, original shapeLoRA-Norm’s0.42170.571.12
Flattenoriginal0.18671.900.55
LoRA-NormLoRA-Norm’s0.09772.080.34

Performance follows functional energy, which confirms that magnitude is the pathway. The table also shows why rebalancing is the method. At an unchanged Frobenius norm, flattening removes 81% of the functional energy, whereas scaling the original shape down to LoRA-Norm’s Frobenius norm removes 58% (flattening gives the lower energy on 18/18 checkpoints). Dominant directions are the strongly activated ones, so a norm-based scalar cannot reach the same functional strength. LoRA-Norm reaches it from the weights alone, with no data and no tuned coefficient, and the data-free scalar control (Scalar-F) is significantly worse on all three metrics (+1.44* / +1.23* / −0.77* in favor of LoRA-Norm).

4. Why the Frobenius norm in Q2. Reviewer 8Euc asked whether the gains are attributable to “reduced update magnitude”. The Frobenius norm is the standard data-free measure of update magnitude and the quantity in Eq. 7, so that factorial answers which operation on the weights produces the gain. Section 5.2 answers through which functional quantity it does so. The table above joins the two views, and we will include it in the revision and label Q2’s decomposition as parameter-space.

5. Shape retains a significant effect at matched functional energy. LoRA-Norm forgets less than Scalar-E (FG −0.18 [−0.35, −0.04]), and so does Flat-E (−0.13 [−0.27, −0.02]).

On your proposed reading. We agree that the benefit is mediated mainly by per-module functional strength; this is the conclusion of Section 5 and of our reply to Reviewer FzQ5. We would add one qualification: rebalancing is not an incidental way to pick coefficients. It is the data-free operation that produces the required functional reduction, and it retains a small retention advantage over the oracle that copies its strength.

On the title, abstract and introduction. We believe they are accurate: they name the operation (fix directions, rebalance gains, restore the nuclear norm), and the abstract and introduction already report that energy-matched controls recover much of the benefit. To remove any impression of two accounts, we will state the coupling of Eq. 7 at the start of the introduction and add one sentence to the abstract: rebalancing acts largely by attenuating each module’s functional update strength.

Kind regards,
The Authors

Thanks for the detailed reply.

I generally agree with the view presented in the reply and find it more grounded and true to the results. I would still argue that the abstract and the beginning of the introduction paint a different picture: “imbalance is bad”, rather than “the update is not scaled properly, and rebalancing is an effective way to fix that”. The abstract does not mention the scaling effect at all, while the introduction mentions it only late, after presenting all the imbalance motivation. This sets incorrect expectations for readers and can mislead those who only skim the paper. That was my own experience, and the reviewers’ related concerns suggest the same. I would therefore recommend that the authors revise the abstract and introduction so that they better reflect the paper’s results from the start.

Response to AC 9kac

Dear Area Chair,

Thank you for the follow-up. We agree with your assessment: the abstract and the opening of the introduction led with "imbalance is bad" and mentioned the scaling effect too late. We have revised both, and the updated PDF is uploaded.

Abstract. It now states from the second sentence that trained updates are imbalanced and overly strong, and it rephrases the central claim as "learning where to adapt does not ensure that adaptation strength is well calibrated." It also reports that scaling the original spectrum to the same energy recovers most of the benefit, and closes with: "rebalancing is a data-free way to set update strength, not an end in itself."

Introduction. The second paragraph now introduces over-strength alongside imbalance, and the central argument is restated in terms of strength calibration. The method paragraph states explicitly that a flatter spectrum at fixed nuclear norm is a weaker update (Eq. 7), and the analysis paragraph, the Figure 1 caption and the contributions follow the same account.

We believe the paper now sets the right expectation from the start: the update is not scaled properly, and rebalancing is an effective data-free way to fix that. Thank you for pushing us toward a framing that is more faithful to our own results.

Kind regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.