- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes LoRA Norm, a post-training method for editing completed LoRA adapters without calibration data or additional training. It preserves the updates learned singular directions, transforms its singular values using a fixed hybrid Newton–Schulz procedure, and restores each module’s original nuclear norm.
The edited adapter retains the original rank and inference structure.
Evaluation covers 18 rank-16 adapters spanning two 8B backbones, three adaptation tasks, and three training runs per model–task pair. Across these checkpoints, average target performance increases from 68.00% to 72.08%, off-task performance increases from 69.55% to 72.79%, and the forgetting gap decreases from 2.71 to 0.34 percentage points. Comparisons include spectral pruning, gradient-guided editing, and several normalization controls.
The analysis shows that normalization also substantially reduces functional update energy. Energy-matched scalar controls reproduce much of the benefit, while stronger functional equalization provides no consistent additional improvement. Consequently, the evidence supports a useful post-training normalization procedure, with limited evidence for an independent advantage from spectral balancing itself.
I like the approach and the simplicity of the idea, and the fact that it does not add overhead.
Critical strengths
- The authors present a practical intervention with consistent (average) benefits. The method uses only adapter parameters, requires no further optimization, and preserves the inference structure.
b. Target performance improves on the majority of checkpoints, and off-task performance also usually improves. Simultaneous average improvements in specialization and retention make this a potentially useful deployment procedure.
c. The analysis is useful and helps clarify the results --- Scalar-E recovers much of the improvement without changing each module's participation ratio, while stronger functional equalization does not consistently improve performance.
This suggests that reducing the update strength contributes to the gains, and that making the directional contributions more uniform is not necessarily better.
In short, the paper presents a simple method with no inference overhead, demonstrates practical improvements, and provides a nice analysis. I therefore lean toward acceptance.
Critical weaknesses
-
The advantage of the specific hybrid transformation over simpler alternatives is still unclear to me. Flat-Nuclear is also data-free and is only 0.15 percentage points behind LoRA-Norm on target performance (Table 4), but its off-task and forgetting results are not reported in that comparison (which is very importnant). Scalar-E also achieves a higher average target score, although the difference is not statistically significant, it requires activation data, and LoRA-Norm has a lower forgetting gap (Tables 17 and 22). The authors acknowledge these limitations. Still, while the results support adjusting gains after training, it is less clear when this particular transformation is preferable.
-
Reconstruction itself changes some results, but this is not fully controlled in the main evaluation. Appendixes shows that reconstructing the adapter without changing its singular values improves target performance by about 0.47 percentage points on average across six archived checkpoints, and that different factorizations can change predictions. However, the same reconstruction-only control is not reported for the main 18 checkpoints. This does not invalidate the overall gains, but it makes it harder to separate small improvements from numerical effects introduced by reconstruction.
Questions for authors
-
Can you evaluate Flat-Nuclear using the same setup as the main results and report Target, Off, and FG with paired confidence intervals? This would help clarify whether HNS provides a meaningful benefit over exact flattening.
-
Can you report Target, Off, and FG for reconstruction alone on the main 18 checkpoints, without changing the singular values and using the same reconstruction and inference setup as LoRA-Norm? Do the conclusions remain the same when comparing against this baseline?
-
Could you also compare with the
smooth_abspolicy from Spectral Surgery? The original paper describes it as a robust default and reports large performance drops forgrad_directionon some tasks. Would including it change the conclusions from Table 2?
Could you also report LoRA-Norm restricted to o_proj and down_proj, using the same checkpoints and evaluation setup? This would help clarify whether its advantage over Spectral Surgery remains when both methods edit the same module families.
Discussion deadline approaching
Dear Reviewers and Authors,
Thank you for providing detailed reviews and rebuttal.
To the authors: After carefully reading the reviews and rebuttal, I have a clarification question for the authors regarding the paper’s position on the main source of LoRA-Norm’s gains.
The title, abstract, beginning of the introduction, and the reply to Reviewer 8Euc (Q2) present spectral rebalancing as the main contributor. However, the end of the introduction and Section 5 indicate that energy-matched scaling with the original spectrum shape (Scalar-E) recovers most of the benefit.
So what is the main contributor: shape or magnitude? Does Scalar-E correspond to “original shape + LoRA-Norm’s magnitude (in terms of functional energy)”? Why is the Frobenius norm, rather than functional energy, used as the magnitude axis in the analysis in Q2? Would it be fair to say that LoRA-Norm works mainly by reducing per-module magnitude, with rebalancing being a data-free way to find suitable reduction coefficients rather than a necessary ingredient in itself? If so, would you align the title, abstract and introduction with this view?
Kind regards,
AC