- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper employs a variety of techniques to control LoRA-based LLM fine-tuning, including the choice of the scaling factor and the shrinkage of matrix A. The paper proposes using a linear alpha-schedule to set the value of the parameter α, and introduces a new regularization term to constrain the parameter optimization process. The paper provides many experimental results, including the scores of LLMs of different sizes, such as 0.5B, 1.5B, 3B, 7B, and 14B, on the GSM8K benchmark. These results show that, on most of the models used in the paper, fine-tuning with the paper's method can achieve accuracy on GSM8K that surpasses almost all compared methods, including some classic baselines and recent work (Stable-LoRA). In addition, the paper also provides many ablation experimental results, and these results show that the Corrected subspace anchor technique used in the paper is effective.
Critical strengths
The paper is well illustrated. The figures are logically organized and clearly presented, which makes the paper easy to follow and helps readers understand the key ideas and experimental setups intuitively. The paper includes a substantial amount of experimental work. The results are rich, covering a wide range of settings and baselines. In addition, the authors conduct detailed ablation studies, which help to isolate the contribution of individual components and provide a clearer picture of how each technique affects performance.
Critical weaknesses
This paper does not provide a good description of the problem it studies. The paper claims to study robust scaling, but it does not give relevant definitions, for example, how to measure stability, what stable tuning is, and what unstable tuning is (no experimental phenomena or theoretical analysis are provided). Even after reading the entire paper, I am still not clear which factor this paper focuses on regarding its influence on the fine-tuning process. The method is not substantial enough, and the innovation is relatively minor. In the Method section, this paper lists many techniques, however, Rank-stabilized scaling was proposed by Kalajdzievski (2023), A-side multiplicative shrinkage was proposed by Wu et al. (2026), and the 4-bit-aware implementation does not seem to be very clearly related to this paper. The paper does not elaborate in detail on the two techniques proposed by the authors, namely Linear α-schedule and Corrected subspace anchor, and after reading the paper I still do not understand why these two techniques are effective or how they work.
Questions for authors
The paper claims to study robust scaling, but it does not give relevant definitions. What exactly does the unstable tuning process refer to? How do you measure stability of a tuning process? What impact do Rank-stabilized scaling and Corrected subspace anchor have on the tuning process? Why do they promote the formation of robust scaling? Is the rank given in Table 2 the same rank used to obtain the results in Table 1? The experiments are almost all conducted on outdated models. Are there also results on models released within the past year (such as Qwen3, Qwen3.5, etc.)?
6 comments
[Part 2/5] What rank-stabilised scaling does to the tuning process
Mechanism. Vanilla LoRA scales the update ΔW = BA by s = α/r. At a fixed α, the adapter's contribution and its gradient scale therefore shrink as the rank grows, so an α tuned at one rank is wrong at another. rsLoRA instead uses s = α/√r, which keeps the update's magnitude independent of rank. In this sense it can support robust scaling, but specifically robustness to rank, not to model size.
Our measurement. α=48 was tuned at r=16. As the rank moves away from 16, both methods drift away from their tuned effective scale, and rsLoRA drifts less:
| rank | vanilla s = α/r | ours s = α/√r | vanilla drift | ours drift |
|---|---|---|---|---|
| 8 | 6.000 | 16.971 | 2.00× | 1.41× |
| 16 | 3.000 | 12.000 | 1.00× | 1.00× |
| 32 | 1.500 | 8.485 | 0.50× | 0.71× |
| 64 | 0.750 | 6.000 | 0.25× | 0.50× |
The scaling therefore does what it is designed to do. Its accuracy effect at the submitted settings is not positive, however:
- Ladder rung. The
lora → rslorarung is −0.86 pp at 1.5B and −1.04 pp at 7B (3 seeds). - Rank sweep (1.5B, seed 42). Vanilla is ahead at every rank: 61.26 / 61.18 / 61.41 / 61.49 for vanilla against 60.05 / 60.42 / 60.35 / 57.77 for ours.
Why. A plausible explanation is a side effect of the scaling: at r=16 and α=48, rsLoRA's s=12 is four times vanilla's s=3. Under Adam, the per-step change in ΔW grows with s, so at the same learning rate the method takes larger effective steps. This is consistent with the small cost at lr=1e-4. At lr=3e-4 a post-hoc test confirms that it pushes some runs into collapse (Part 4).
So the interaction between rsLoRA scaling and the learning rate is the stability factor that matters in this recipe. We will state this in the revision instead of presenting rsLoRA as a stabiliser. rsLoRA itself is prior work (Kalajdzievski, 2023), and so is A-side shrinkage (Wu et al., 2026). We will present both as borrowed components with this measured behaviour, not as contributions of ours.
A-side shrinkage, for completeness. It is meant to keep ‖A‖ from growing against ‖B‖ at high learning rates. At lr ≤ 1e-4 its gate never fires (112/112 layers stay active), so in practice it acts as a fixed exponential decay on A. The measured ‖A‖ trajectory matches pure geometric decay to 98.8%. The gate engages only at lr ≥ 3e-4. We will describe the component accordingly.
[Part 3/5] What the corrected subspace anchor does, and why it matters
Design. The anchor penalises the part of the update that lies in W₀'s top-k singular subspace:
L_anchor = λ/(k·r) · Σ‖U_kᵀ B A V_k‖².
Here U_k and V_k are W₀'s leading singular vectors. Those directions carry the pretrained model's dominant features, and the intent is to adapt in the remaining directions so that these features are protected.
Why the 4-bit implementation is part of this paper. The anchor needs W₀'s singular vectors. Under QLoRA, W₀ exists only as NF4 blocks, and PEFT wraps those layers as Linear4bit, which a standard implementation skips silently. Our own early version covered 0 of 112 layers and raised no error. Dequantising W₀ before the SVD ("corrected") is what makes the anchor present at all on the 7B and 14B backbones. It is a precondition for using the recipe across scales, not a separate idea.
First finding: anchor strength must be set by gradient share. In one backward pass we measured the anchor's share of the total gradient. At the submitted λ=5e-4 it is 0.000%: the task gradient norm is 6.80e-01, against an anchor gradient of 1.62e-05 at λ=1. The λ that gives a 10% share differs across backbones:
- 4.19e3 for Qwen2-1.5B;
- 7.0e2 for Qwen2.5-7B;
- 3.16e3 for Qwen3-1.7B;
- 1.66e3 for Qwen3.5-2B.
This is a concrete scale-dependent factor: a fixed raw λ gives very different regularisation strengths at different scales. The quantity that transfers is the gradient share.
Second finding: once it acts, the anchor does exactly what it is designed to do. We set λ per backbone by a 10%-share rule fixed before any run, and we report every seed. We then measured the fraction of update energy in W₀'s top-64 subspace, ‖U₆₄ᵀ ΔW V₆₄‖² / ‖ΔW‖², averaged over 112 layers:
| 1.5B | 7B | |
|---|---|---|
| vanilla LoRA | 0.691% | 0.218% |
| submitted λ (inert) | 0.682% | 0.188% |
| anchor at ~1% share | 0.316% | 0.128% |
| anchor at ~10% share | 0.159% | 0.066% |
| random-subspace reference | 0.608% | 0.128% |
This is the effect on the tuning process. The acting anchor steers the update out of the pretrained principal directions, in proportion to its gradient share:
- It removes 77% (1.5B) and 70% (7B) of the energy in that subspace.
- Across 3 seeds, the ranges do not overlap with those of any other arm.
- Vanilla LoRA concentrates its update above the random reference, 0.69% against 0.61%, which the acting anchor reverses.
- No other component (rsLoRA, shrinkage, ramp) moves the fraction below the reference.
Third finding: the accuracy consequence. We fixed the seed extension before the new runs (1.5B n=9, 7B n=5). We scored GSM8K and held-out SVAMP, which was never used for any choice. Only λ differs between the compared arms:
| acting − submitted | GSM8K | SVAMP | per-problem, both datasets |
|---|---|---|---|
| 1.5B, n=9 | +0.35 | +0.51, 9/9 seeds | McNemar p = 0.012 |
| 7B, n=5 | +0.33 | +0.16 | p = 0.076 |
The Fisher-combined result is p = 0.007. Against vanilla LoRA, the acting anchor reaches statistical parity over both datasets (−0.10 / −0.06 pp), where the submitted configuration was behind. On non-math transfer (Alpaca → 4 tasks, 6 seeds) it is +0.26 pp over vanilla at 5 of 6 seeds, and +0.11 pp over the submitted configuration at 6 of 6 seeds.
[Part 1/5] Summary; definition of the problem and of stability
We thank the reviewer for the careful reading. The central criticism is fair: the submission uses "robust scaling" and "stability" without defining them. Below we give the definitions and measure every one of them. We also explain the mechanism of rsLoRA scaling and of the corrected anchor, answer the rank question, and add two recent backbones. All of this rests on 159 new training runs with 0 failures. In summary:
What problem the paper studies. The setting is one LoRA configuration reused across backbone sizes without retuning. Two things matter in that setting: does training stay well-behaved, and does accuracy stay competitive? The factor we focus on is how much of the update lands in the pretrained model's principal singular directions, together with how strongly the regulariser controls it at each scale. Part 3 shows this is measurable and scale-dependent.
Definition of stability: four layers, each reported separately
"Unstable tuning" therefore means failing layer 1, 2 or 3, and "non-robust scaling" means a poor result on layer 4. The layers can disagree. For example, a run can never diverge, so layer 1 is clean, yet still be highly seed-sensitive, so layer 4 is poor. The submission conflated the four under one word.
Measurements:
Conclusion. Measured this way, the method is as stable as vanilla LoRA on layers 1, 2 and 4. We will claim "no instability in either method", not "more stable".