acceptodds
Under review as a conference paper at ICLR 2027

Stable-LoRA+: Regularizing Low-Rank Adaptation for Robust Scalin

Abstract

The efficiency of low-rank adaptation does not ensure stable training as language models increase in size. Settings that are effective at one scale may degrade accuracy or lead to divergence at another. We study robust scaling of adaptation for mathematical reasoning, evaluating training stability and predictive accuracy across model sizes under a fixed training configuration, separately from performance after tuning. We introduce Stable-LoRA⁺, a training method that regularizes the factorized adaptation process. Its premise is that scaling the adapter output alone does not fully constrain factor evolution. The method therefore controls factor growth during training while penalizing update components aligned with the pretrained weight's leading singular subspaces. The same regularization is explicitly implemented for 4-bit backbones. Across more than 130 training-and-evaluation runs on five Qwen backbones (0.5B–14B), the strict fixed-configuration recipe matches vanilla LoRA's macro-average GSM8K accuracy (69.55%). With protocol and seed selection, the broader best-protocol comparison reaches 70.14% and mean rank 1.50. At lr = 10⁻³, vanilla LoRA diverges on 3B/7B, losing at least 76 percentage points, while our shared 10⁻⁴ configuration remains stable across the evaluated backbones. A quantization audit also identifies a silently inactive anchor, whose correction recovers 1.36 points at 7B. These results motivate regularization of factorized updates as an approach to robust adaptation across the evaluated model scales.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
Training Stability, Low-Rank Adaptation, Cross-Scale Robustness

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_miSsOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper employs a variety of techniques to control LoRA-based LLM fine-tuning, including the choice of the scaling factor and the shrinkage of matrix A. The paper proposes using a linear alpha-schedule to set the value of the parameter α, and introduces a new regularization term to constrain the parameter optimization process. The paper provides many experimental results, including the scores of LLMs of different sizes, such as 0.5B, 1.5B, 3B, 7B, and 14B, on the GSM8K benchmark. These results show that, on most of the models used in the paper, fine-tuning with the paper's method can achieve accuracy on GSM8K that surpasses almost all compared methods, including some classic baselines and recent work (Stable-LoRA). In addition, the paper also provides many ablation experimental results, and these results show that the Corrected subspace anchor technique used in the paper is effective.

Critical strengths

The paper is well illustrated. The figures are logically organized and clearly presented, which makes the paper easy to follow and helps readers understand the key ideas and experimental setups intuitively. The paper includes a substantial amount of experimental work. The results are rich, covering a wide range of settings and baselines. In addition, the authors conduct detailed ablation studies, which help to isolate the contribution of individual components and provide a clearer picture of how each technique affects performance.

Critical weaknesses

This paper does not provide a good description of the problem it studies. The paper claims to study robust scaling, but it does not give relevant definitions, for example, how to measure stability, what stable tuning is, and what unstable tuning is (no experimental phenomena or theoretical analysis are provided). Even after reading the entire paper, I am still not clear which factor this paper focuses on regarding its influence on the fine-tuning process. The method is not substantial enough, and the innovation is relatively minor. In the Method section, this paper lists many techniques, however, Rank-stabilized scaling was proposed by Kalajdzievski (2023), A-side multiplicative shrinkage was proposed by Wu et al. (2026), and the 4-bit-aware implementation does not seem to be very clearly related to this paper. The paper does not elaborate in detail on the two techniques proposed by the authors, namely Linear α-schedule and Corrected subspace anchor, and after reading the paper I still do not understand why these two techniques are effective or how they work.

Questions for authors

The paper claims to study robust scaling, but it does not give relevant definitions. What exactly does the unstable tuning process refer to? How do you measure stability of a tuning process? What impact do Rank-stabilized scaling and Corrected subspace anchor have on the tuning process? Why do they promote the formation of robust scaling? Is the rank given in Table 2 the same rank used to obtain the results in Table 1? The experiments are almost all conducted on outdated models. Are there also results on models released within the past year (such as Qwen3, Qwen3.5, etc.)?

6 comments

[Part 1/5] Summary; definition of the problem and of stability

We thank the reviewer for the careful reading. The central criticism is fair: the submission uses "robust scaling" and "stability" without defining them. Below we give the definitions and measure every one of them. We also explain the mechanism of rsLoRA scaling and of the corrected anchor, answer the rank question, and add two recent backbones. All of this rests on 159 new training runs with 0 failures. In summary:

  1. Definitions. "Stability" is now defined as four separately reported, measurable layers, set out below.
  2. Mechanism. We measured directly what the corrected anchor does to the tuning process. Once it carries enough gradient to act, it moves 70–77% of the update energy out of W₀'s top-64 singular subspace, and no other component does this. It also improves accuracy over the submitted configuration (Fisher-combined p = 0.007).
  3. Rank. Table 2 reports the per-backbone ranking of the methods by their Table 1 accuracies, not a LoRA rank. Every run behind both tables uses the same LoRA rank, r = 16.
  4. Recent models. We added Qwen3-1.7B-Base and Qwen3.5-2B-Base.

What problem the paper studies. The setting is one LoRA configuration reused across backbone sizes without retuning. Two things matter in that setting: does training stay well-behaved, and does accuracy stay competitive? The factor we focus on is how much of the update lands in the pretrained model's principal singular directions, together with how strongly the regulariser controls it at each scale. Part 3 shows this is measurable and scale-dependent.

Definition of stability: four layers, each reported separately

  1. Hard failure. A non-finite loss or gradient, or a run that does not finish. Our trainer counts non-finite microbatches in every run.
  2. Trajectory anomaly. A loss spike (an upward step-to-step change above 2σ of all step changes), or ‖A‖/‖B‖ growth outside a stated band. Both are logged every 25 steps.
  3. Endpoint collapse. Final accuracy far below the arm's own seed mean; in practice, accuracy near 0 with training loss above 5.
  4. Sensitivity. The spread of final accuracy across seed, learning rate, rank and backbone under one fixed configuration. This is the layer that "robust scaling" refers to.

"Unstable tuning" therefore means failing layer 1, 2 or 3, and "non-robust scaling" means a poor result on layer 4. The layers can disagree. For example, a run can never diverge, so layer 1 is clean, yet still be highly seed-sensitive, so layer 4 is poor. The submission conflated the four under one word.

Measurements:

  • Layer 1. 0 non-finite microbatches in all 159 runs, across every arm, backbone (Qwen2-1.5B, Qwen2.5-7B, Qwen3-1.7B, Qwen3.5-2B), rank and learning rate. This includes the NF4 dequantisation path at 7B.
  • Layer 2. Loss-curve roughness (vanilla vs ours) is 0.1807 vs 0.1829 at 7B and 0.2778 vs 0.2773 at 1.5B. Neither arm has a spike above 2σ at either scale, at the logged resolution. Shrinkage behaves as designed: ‖A‖ goes 2.31 → 1.71 with it, while vanilla's ‖A‖ rises slightly.
  • Layer 3. Both methods collapse at lr=1e-3. At 7B and lr=3e-4, the method's two arms (submitted and acting anchor) also collapse in 2 of 4 runs, against 0 of 2 for vanilla. Part 4 gives the cause.
  • Layer 4. Seed spread is similar: at n=3 ours is tighter at 7B (0.27 vs 0.42) and 1.5B (0.62 vs 0.72); with extended seeds it is 0.75 vs 0.41 at 7B and 0.63 vs 0.66 at 1.5B. A pooled Brown-Forsythe test gives p = 0.926.

Conclusion. Measured this way, the method is as stable as vanilla LoRA on layers 1, 2 and 4. We will claim "no instability in either method", not "more stable".

[Part 2/5] What rank-stabilised scaling does to the tuning process

Mechanism. Vanilla LoRA scales the update ΔW = BA by s = α/r. At a fixed α, the adapter's contribution and its gradient scale therefore shrink as the rank grows, so an α tuned at one rank is wrong at another. rsLoRA instead uses s = α/√r, which keeps the update's magnitude independent of rank. In this sense it can support robust scaling, but specifically robustness to rank, not to model size.

Our measurement. α=48 was tuned at r=16. As the rank moves away from 16, both methods drift away from their tuned effective scale, and rsLoRA drifts less:

rankvanilla s = α/rours s = α/√rvanilla driftours drift
86.00016.9712.00×1.41×
163.00012.0001.00×1.00×
321.5008.4850.50×0.71×
640.7506.0000.25×0.50×

The scaling therefore does what it is designed to do. Its accuracy effect at the submitted settings is not positive, however:

  • Ladder rung. The lora → rslora rung is −0.86 pp at 1.5B and −1.04 pp at 7B (3 seeds).
  • Rank sweep (1.5B, seed 42). Vanilla is ahead at every rank: 61.26 / 61.18 / 61.41 / 61.49 for vanilla against 60.05 / 60.42 / 60.35 / 57.77 for ours.

Why. A plausible explanation is a side effect of the scaling: at r=16 and α=48, rsLoRA's s=12 is four times vanilla's s=3. Under Adam, the per-step change in ΔW grows with s, so at the same learning rate the method takes larger effective steps. This is consistent with the small cost at lr=1e-4. At lr=3e-4 a post-hoc test confirms that it pushes some runs into collapse (Part 4).

So the interaction between rsLoRA scaling and the learning rate is the stability factor that matters in this recipe. We will state this in the revision instead of presenting rsLoRA as a stabiliser. rsLoRA itself is prior work (Kalajdzievski, 2023), and so is A-side shrinkage (Wu et al., 2026). We will present both as borrowed components with this measured behaviour, not as contributions of ours.

A-side shrinkage, for completeness. It is meant to keep ‖A‖ from growing against ‖B‖ at high learning rates. At lr ≤ 1e-4 its gate never fires (112/112 layers stay active), so in practice it acts as a fixed exponential decay on A. The measured ‖A‖ trajectory matches pure geometric decay to 98.8%. The gate engages only at lr ≥ 3e-4. We will describe the component accordingly.

[Part 3/5] What the corrected subspace anchor does, and why it matters

Design. The anchor penalises the part of the update that lies in W₀'s top-k singular subspace:

L_anchor = λ/(k·r) · Σ‖U_kᵀ B A V_k‖².

Here U_k and V_k are W₀'s leading singular vectors. Those directions carry the pretrained model's dominant features, and the intent is to adapt in the remaining directions so that these features are protected.

Why the 4-bit implementation is part of this paper. The anchor needs W₀'s singular vectors. Under QLoRA, W₀ exists only as NF4 blocks, and PEFT wraps those layers as Linear4bit, which a standard implementation skips silently. Our own early version covered 0 of 112 layers and raised no error. Dequantising W₀ before the SVD ("corrected") is what makes the anchor present at all on the 7B and 14B backbones. It is a precondition for using the recipe across scales, not a separate idea.

First finding: anchor strength must be set by gradient share. In one backward pass we measured the anchor's share of the total gradient. At the submitted λ=5e-4 it is 0.000%: the task gradient norm is 6.80e-01, against an anchor gradient of 1.62e-05 at λ=1. The λ that gives a 10% share differs across backbones:

  • 4.19e3 for Qwen2-1.5B;
  • 7.0e2 for Qwen2.5-7B;
  • 3.16e3 for Qwen3-1.7B;
  • 1.66e3 for Qwen3.5-2B.

This is a concrete scale-dependent factor: a fixed raw λ gives very different regularisation strengths at different scales. The quantity that transfers is the gradient share.

Second finding: once it acts, the anchor does exactly what it is designed to do. We set λ per backbone by a 10%-share rule fixed before any run, and we report every seed. We then measured the fraction of update energy in W₀'s top-64 subspace, ‖U₆₄ᵀ ΔW V₆₄‖² / ‖ΔW‖², averaged over 112 layers:

1.5B7B
vanilla LoRA0.691%0.218%
submitted λ (inert)0.682%0.188%
anchor at ~1% share0.316%0.128%
anchor at ~10% share0.159%0.066%
random-subspace reference0.608%0.128%

This is the effect on the tuning process. The acting anchor steers the update out of the pretrained principal directions, in proportion to its gradient share:

  • It removes 77% (1.5B) and 70% (7B) of the energy in that subspace.
  • Across 3 seeds, the ranges do not overlap with those of any other arm.
  • Vanilla LoRA concentrates its update above the random reference, 0.69% against 0.61%, which the acting anchor reverses.
  • No other component (rsLoRA, shrinkage, ramp) moves the fraction below the reference.

Third finding: the accuracy consequence. We fixed the seed extension before the new runs (1.5B n=9, 7B n=5). We scored GSM8K and held-out SVAMP, which was never used for any choice. Only λ differs between the compared arms:

acting − submittedGSM8KSVAMPper-problem, both datasets
1.5B, n=9+0.35+0.51, 9/9 seedsMcNemar p = 0.012
7B, n=5+0.33+0.16p = 0.076

The Fisher-combined result is p = 0.007. Against vanilla LoRA, the acting anchor reaches statistical parity over both datasets (−0.10 / −0.06 pp), where the submitted configuration was behind. On non-math transfer (Alpaca → 4 tasks, 6 seeds) it is +0.26 pp over vanilla at 5 of 6 seeds, and +0.11 pp over the submitted configuration at 6 of 6 seeds.

Reviewer_reunOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

Stable-LoRA+ studies robust scaling: reusing one low-rank adaptation configuration across backbone sizes while maintaining training stability and predictive accuracy. It combines rank-stabilized output scaling, a linear amplitude schedule, gated shrinkage of the down-projection factor AA, and a penalty on the unscaled update BABA projected into pretrained singular subspaces. For NF4 backbones, it constructs those subspaces from dequantized weights. Experiments on MetaMathQA/GSM8K cover five Qwen backbones from 0.5B to 14B, with repeated seeds, component ablations, and training diagnostics.

Critical strengths
  1. The NF4 audit establishes a concrete empirical effect. Correctly detecting and dequantizing Linear4bit modules activates an otherwise inactive anchor. The three paired 7B gains are 1.22, 1.29, and 1.59 percentage points, with a reported mean of 1.36 points (Tables 4/7). The near-zero 14B effect helps delimit this finding. The magnitude and scale dependence of this effect require measurement and are useful observations.

  2. The proposed configuration is evaluated beyond a single result. Fifteen fixed-recipe runs finish across five scales with the same trainable-parameter budget as LoRA and comparable reported runtime. The 1.5B rank sweep improves over vanilla LoRA by 0.6–1.5 points across five ranks; schedule and asymmetric-shrinkage ablations support the chosen local settings.

Critical weaknesses
  1. The remaining methodological idea is mainly a configuration of existing controls. Rank scaling comes from rsLoRA [1], and gated AA shrinkage from Stable-LoRA [2]. The RPSFT preprint explicitly gives the projected-update penalty for BABA in Appendix A.2 [3]. That optional LoRA variant was not experimentally evaluated, so implementing and testing the scheduled NF4 recipe is an addition. However, dequantizing the base weight makes the existing objective computable; it introduces no new constraint or information. The main remaining design is the fixed 32-to-48 amplitude ramp alongside these controls. Reciprocal rescaling explains why factor norms and output updates differ, but does not determine this schedule or establish why this particular combination enables cross-scale reuse. The local ablations support settings, while the normalized dynamics describe their behavior; they do not establish a new relationship between backbone scale and the required controls.

  2. The complete-method results do not establish a substantial practical advance on robust scaling. Canonical seed-42 changes against default LoRA are −1.13, +1.74, −0.45, −0.60, and +0.45 points across the five scales; both macro-averages round to 69.55%. The stronger 70.14% aggregate uses the disclosed protocol and seed selection. The paired 7B anchor effect is an internal component comparison, while the complete recipe's reported seed-42 accuracy remains below default LoRA there. Default LoRA also works across all five scales at 10−410^{-4}; its failures at 10−310^{-3} do not establish a reliability advantage for this recipe at 10−410^{-4}. Thus, the integration supplies local measurements without demonstrating a distinctive operating capability or reduced configuration-selection cost.

References

[1] Damjan Kalajdzievski, "A Rank Stabilization Scaling Factor for Fine-Tuning with LoRA," arXiv:2312.03732, 2023. Paper

[2] Yize Wu, Ke Gao, Ling Li, and Yanjun Wu, "Stable-LoRA: Stabilizing Feature Learning of Low-Rank Adaptation," ICLR, 2026. Paper

[3] Hangzhan Jin, Tianwei Ni, Lu Li, Pierre-Luc Bacon, Mohammad Hamdaqa, and Doina Precup, "Rotation-Preserving Supervised Fine-Tuning," arXiv:2605.10973, 2026. Paper

Questions for authors
  1. Clarification: the 7B comparison. Appendix C reports 82.94±0.28 for “rsLoRA+StableLora,” while Tables 4/7 report 81.53 for anchor-off and 82.89 for the full recipe. How does the Appendix C configuration differ from these runs, particularly in its amplitude schedule and training protocol, and are the seeds matched? This determines which existing comparison isolates the benefit of the added controls over the simpler combination, and how to interpret the reported 1.36-point anchor gain.

  2. Clarification: selection of the shared configuration. Figure 4 reports local sweeps, but how were the amplitude schedule, shrinkage, and anchor settings selected before fixing the configuration used in Tables 3/7? Please identify the model scales and evaluation split used for those choices, including whether 3B/7B/14B outcomes informed them. This would clarify how much the fixed-recipe results support reuse on backbones outside configuration selection, which is central to the robust-scaling contribution.

8 comments

[Part 1/5] Summary of our response; Q1 (Appendix C vs. Tables 4/7)

We thank the reviewer for a precise and careful review. Both questions have definite answers in our archived artifacts. Answering them led us to rerun the comparisons under fully matched controls: 159 new training runs, 52 re-evaluations of the archived adapters, 77 held-out/transfer re-scores, and 0 failed runs. We summarise first and then answer each point.

Summary

  1. Q1. The seeds are matched. The reported 1.36-point anchor gain, however, is a precision confound: the anchor-on arm was evaluated in bf16 and the anchor-off arm in NF4. We withdraw the claim.
  2. Q2. α and the ramp were selected on 0.5B and 1.5B, and λ on 3B, using GSM8K test accuracy. Only 7B and 14B are genuine transfer scales. We will state this in the main text and add a held-out split (SVAMP).
  3. W1. At the submitted λ the anchor contributes 0.000% of the gradient. Once λ is calibrated by the anchor's gradient share, the anchor acts. It then removes 77% (1.5B) and 70% (7B) of the update energy from W₀'s top-64 singular subspace, which no other component does. It also improves over the submitted configuration (per-problem test on GSM8K+SVAMP, Fisher-combined p = 0.007). The gradient share is the scale-transferable quantity the reviewer asks for.
  4. W2. We agree with the arithmetic. With the anchor acting, the method reaches statistical parity with vanilla LoRA on math, not superiority, and we will reframe the paper around what is measured.

Q1. How do 82.94±0.28, 81.53 and 82.89 relate, and are the seeds matched?

All three are Qwen2.5-7B, r=16, lr=1e-4, 10K MetaMathQA, 1 epoch, NF4 training, seeds {42, 43, 44}. The seeds are matched, three per arm:

Paper figureα during trainingα in saved configEval precisionRe-computed mean ± sd
App. C "rsLoRA+StableLora" 82.9448, constant48bf1682.9416 ± 0.3474
Tab. 4/7 full recipe 82.8932→48 ramp32bf1682.8911 ± 0.1158
Tab. 4/7 anchor-off 81.5332→48 ramp32NF481.5264 ± 0.2870

(The paper's ±0.28 is the population sd; we list sample sd.)

  1. Appendix C vs. the full recipe. The Appendix C arm differs from the full recipe in exactly two ways: α is held at 48 rather than ramped, and there is no anchor. It is the simpler combination. The gap is 82.94 vs. 82.89, a 0.05 pp difference that lies well inside our ±0.53 pp evaluation floor. The reviewer is right: at the submitted settings, the added controls do not measurably improve on the simpler combination.
  2. The 1.36 points are not an isolation of the anchor. The paired differences are 1.21 / 1.29 / 1.59. However, they compare an anchor-on arm evaluated in bf16 with an anchor-off arm evaluated in NF4, and three independent pieces of evidence confirm this:
    • Launch scripts. v76/v77/v78/v81 call the evaluator without --use_4bit, while v120 calls it with the flag.
    • Eval wall-clock. Anchor-off evaluations took 2.33 to 2.41 h, against about 1.0 h for every other 7B arm.
    • Direct re-evaluation. Scoring the same adapter in bf16 instead of NF4 gives +1.30 ± 0.45 pp (n=12), which is 92–96% of the claimed gain.

Part 2 gives the matched decomposition.

[Part 2/5] Q1 continued: matched decomposition and 14B; Q2 (configuration selection)

We re-scored all nine archived scheduled checkpoints under four conditions (34 re-evaluations, 0 failures):

anchor-on − anchor-off (7B, 3 seeds × 3 seeds)mean ± sd
as published (bf16 vs NF4, α=32)+1.21 ± 0.53
matched, NF4, α=32+0.10 ± 0.34
matched, NF4, α=48 (trained α)+0.08 ± 0.77
matched, bf16, α=48−0.05 ± 0.34

We withdraw the 1.36-point claim and its scale-dependence interpretation. We also found a smaller bookkeeping error. save_pretrained stores the construction-time α=32, so the scheduled checkpoints were scored at s=8 although they were trained at s=12. Correcting this is worth +0.42 ± 0.45 pp (n=12). The effect is below the noise floor, and it applies to both arms.

Noise floors. Our re-evaluation reproduces the archived scores within ±7/1319 problems (±0.53 pp). Two accidental replicates, identical in configuration and seed, differ by 0.45 and 1.21 pp because of kernel and dataloader nondeterminism. We will report both floors and label single-run differences below 1.21 pp as unresolved.

14B. The reviewer's observation that the 14B effect is near zero is confirmed under matched conditions (18 re-evaluations):

14B, anchor on − offper seedmeanMcNemar
NF4, α=48+0.45 / −0.76 / 0.00−0.10p = 0.771
bf16, α=48+0.76 / +0.45 / −0.68+0.18p = 0.589

The precision shift behaves differently at the two scales. At 14B, bf16 − NF4 on the same adapter is −0.21 ± 1.21 pp, with signs split 3/3. At 7B it is +1.30 ± 0.45 pp and always positive. That is why a one-sided precision mismatch produced +1.36 at 7B and nothing at 14B: the contrast the paper read as scale dependence is a precision effect.

Why the anchor-off ablation could not matter. At the submitted λ=5e-4 the anchor supplies 0.000% of the gradient. On the 1.5B checkpoint the task gradient norm is 6.80e-01, and the anchor gradient at λ=1 is only 1.62e-05. Removing the anchor therefore removed a term that was not acting. Part 3 shows what changes once it does act.

Q2. How was the shared configuration selected?

We reconstructed the order from the timestamps of all 186 archived runs:

stagebackboneswhat it fixed
1Qwen2-0.5B (~40 runs)α=48
2Qwen2-1.5B sweeps32→48 ramp
3Qwen2.5-3Bλ=5e-4
4Qwen2.5-7B, then 14Bfirst runs after all settings were fixed
5baselines, LR sweeps, anchor-off controlsrun after the configuration froze

So 0.5B and 1.5B selected α and the ramp, and 3B selected λ. 7B and 14B outcomes informed no choice, so only 7B and 14B are genuine transfer scales. Selection used GSM8K test accuracy, because the pipeline has no validation split. We will state all of this in the main text.

Held-out check. Because selection touched GSM8K test, we also scored every adapter on SVAMP (1000 problems). SVAMP was never used for any choice. With the pre-registered seed extension (1.5B n=9, 7B n=5):

  • Acting anchor vs. submitted configuration (1.5B): the acting anchor is ahead at 9 of 9 seeds, by +0.51 ± 0.25 pp (sign-flip p = 0.004).
  • Submitted configuration vs. vanilla: −0.02 at 7B, and −0.91 at 1.5B (McNemar p = 0.004).
  • Acting anchor vs. vanilla: +0.14 at 7B and −0.40 at 1.5B, neither significant.

On the split nothing was tuned on, the configuration selected on GSM8K is no better than vanilla. The one change the mechanism motivates, making the anchor act, is the one that helps.

[Part 3/5] W1: what remains beyond existing controls

We accept the attribution as stated:

  • rank-stabilised scaling comes from rsLoRA (Kalajdzievski, 2023);
  • gated A-side shrinkage comes from Stable-LoRA (Wu et al., 2026);
  • the projected-update penalty comes from RPSFT (Appendix A.2).

We will cite RPSFT explicitly and narrow the contribution. Three things remain, and the new experiments supply the evidence for two of them.

(1) The anchor is computable under 4-bit quantization. As the reviewer says, dequantizing W₀ adds no task information, but it is necessary. PEFT wraps quantized layers as Linear4bit, and an implementation that filters for LoraLinear skips them silently. Our own early version covered 0 of 112 layers without raising an error. Dequantizing W₀ before the SVD is what makes the anchor present in any 7B or 14B run. We will claim this as an implementation contribution, not as a source of accuracy.

(2) A measured relationship between scale and the required control: the gradient share. The reviewer asks for "a new relationship between backbone scale and the required controls", and measuring the anchor's share of the total gradient supplies one. The λ that gives a 10% share is:

backboneλ at 10% share
Qwen2-1.5B4.19e3
Qwen2.5-7B7.0e2
Qwen3-1.7B3.16e3
Qwen3.5-2B1.66e3

At 1.5B it moves only to 5.08e3 when the training data changes from math to Alpaca.

A fixed raw λ therefore cannot give the same regularisation strength at different scales. The submitted λ=5e-4 was about 8 million× too small, and it was inert at both scales where we measured it. We confirmed the inertness independently: a 1000× change of λ inside the original range (5e-4 → 5e-1) moves the adapter weights 0.93× as much as rerunning the identical job. The quantity that transfers is the share, which takes one backward pass to measure.

(3) What the anchor does once it acts. We pre-registered the protocol before any run: λ was fixed per backbone by the 10%-share rule, the seed extensions were fixed in advance, and every seed is reported. The table gives the fraction of update energy lying in W₀'s top-64 singular subspace, ‖U₆₄ᵀ ΔW V₆₄‖² / ‖ΔW‖², with the range over 3 seeds:

1.5B7B
vanilla LoRA0.689–0.694%0.215–0.219%
each other component (rsLoRA, A-shrink, ramp)0.671–0.688%0.174–0.191%
submitted λ (inert)0.677–0.684%0.186–0.191%
anchor at 10% share0.156–0.162%0.066%
random-subspace reference0.608%0.128%

The acting anchor removes 77% / 70% of the update energy from the protected subspace. No other component brings it below the random reference. Vanilla LoRA, rsLoRA and Stable-LoRA do not impose this constraint, and it is now measured rather than asserted.

The accuracy comparison holds the seed, the data subset and the evaluation precision fixed; only λ differs. Acting minus submitted:

nGSM8KSVAMPper-problem, both datasets
1.5B9+0.35 ± 0.67+0.51 ± 0.25 (9/9 seeds)McNemar p = 0.012
7B5+0.33 ± 0.46+0.16 ± 0.17p = 0.076

Fisher-combined across the two scales, p = 0.007.

These results have limits, which we state here:

  • Size. The gains are about a third to half a point.
  • GSM8K alone. On GSM8K alone the gains are not individually significant (p = 0.141 / 0.113).
  • Recent backbones. On Qwen3-1.7B and Qwen3.5-2B the calibrated anchor did not help (−0.48 / −0.61 vs. vanilla, n=3, not significant). We report this as an open question.
  • The 32→48 ramp. Its measured contribution flips sign across seeds, so we keep it as a documented design choice and do not claim it as a contribution.
Reviewer_pipaOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Stable-LoRA+, which introduces regularization into LoRA training to improve stability as the model scales. The method is evaluated on five Qwen models using mathematical reasoning tasks.

Critical strengths
  1. The paper is clearly written and easy to follow.
  2. The experiments on mathematical reasoning tasks are comprehensive.
Critical weaknesses

The proposed method is presented without sufficient motivation or explanation for its individual components. This makes it difficult to understand why these components are necessary and how they contribute to the overall performance.

Questions for authors
  1. Could the authors provide a deeper analysis of why the proposed components work and how each contributes to the overall improvement?
  2. Can the proposed method generalize to other tasks beyond mathematical reasoning? Could the authors provide additional experiments to support its broader applicability?
7 comments

[Part 1/4] Summary; matched component ladder

We thank the reviewer for the positive assessment and for two questions that go to the heart of the method. We answer both with new, matched experiments: 159 new training runs, 0 failures, and a fixed protocol throughout. In brief:

  1. Why the components work. We rebuilt the ablation as a strict ladder that adds one component at a time, under one code path, one data order per seed and one evaluation precision. For each component we measured what it actually does to training, not only its accuracy footprint. The key finding is that the subspace anchor carries the method's protective mechanism. Once it is given enough weight to act, it moves 70–77% of the update energy out of the pretrained principal subspace, and it improves accuracy over the submitted configuration (Fisher-combined p = 0.007).
  2. Beyond math. On a non-mathematical track (Alpaca → ARC-C / HellaSwag / BoolQ / Winogrande, 6 seeds), the method is ahead of matched vanilla LoRA at 5 of 6 seeds. With the anchor acting it is ahead of the submitted configuration at 6 of 6 seeds.

We also report, transparently, where the components do not help.

The matched ladder (GSM8K, 3 seeds each, mean ± sd):

rungrsLoRAA-shrinkα 32→48anchorQwen2-1.5BQwen2.5-7B
lora––––60.50 ± 0.7282.26 ± 0.42
rslora✓–––59.64 ± 1.0881.22 ± 0.99
simplified✓✓––60.35 ± 1.5281.68 ± 0.42
sched✓✓✓–59.97 ± 0.8581.40 ± 0.19
full (submitted λ=5e-4)✓✓✓✓59.99 ± 0.6181.20 ± 0.27

Paired per-rung contributions (same seed, so each delta is the effect of exactly one component):

step1.5B7B
+ rsLoRA−0.86 ± 0.57−1.04 ± 0.94
+ A-shrinkage+0.71 ± 0.82+0.45 ± 0.67
+ α ramp−0.38 ± 1.12−0.28 ± 0.27
+ anchor at submitted λ+0.03 ± 0.90−0.20 ± 0.12
anchor made to act (λ by 10% gradient share), n=3; extended seeds in Part 2+0.56 ± 0.46+0.58 ± 0.39

The last row is the same full recipe, with only λ raised until the anchor carries about 10% of the gradient. Of the rungs this method adds, it is the only one that is positive at all three seeds at both scales. We explain why in Part 2.

Honest reading. At the submitted λ, the full recipe does not beat vanilla LoRA end-to-end: it is −0.51 ± 0.94 pp at 1.5B and −1.06 ± 0.35 pp at 7B. The main reason is mechanistic: at λ=5e-4 the anchor supplies 0.000% of the gradient, so the submitted configuration ran with its main component effectively switched off. Part 2 measures this directly.

[Part 2/4] Q1: why each component should work, and what each does

componentintended mechanismwhat we measured
rsLoRA, s = α/√rkeep the update's magnitude independent of rank, so one α serves every rreduces α drift across rank (0.50× vs 0.25× at r=64); −0.86 / −1.04 pp at r=16
A-side shrinkage γkeep ‖A‖ from growing against ‖B‖ at high learning rates‖A‖ decays 2.31 → 1.71; at lr ≤ 1e-4 the gate never fires, so 98.8% of the effect is plain decay on A; it engages at lr ≥ 3e-4
α ramp 32→48small steps early while B≈0, full scale laterthe effect flips sign across seeds; not resolvable at n=3
subspace anchorkeep ΔW out of W₀'s top-k singular directions, protecting pretrained featuressee below

The anchor: what it does once it acts. The penalty is λ/(k·r)·Σ‖U_kᵀ B A V_k‖². We measured the anchor's share of the total gradient in one backward pass. At the submitted λ=5e-4 the share is 0.000%: the task gradient norm is 6.80e-01, while the anchor gradient at λ=1 is only 1.62e-05. A 10% share needs λ ≈ 4.19e3 at 1.5B and 7.0e2 at 7B.

So the original ablation measured a component that was switched on but not acting. We then pre-registered a rule before any run: λ is set per backbone to give a ~10% gradient share, and every seed is reported. Under that rule we measured the fraction of update energy that lies in W₀'s top-64 singular subspace, ‖U₆₄ᵀ ΔW V₆₄‖² / ‖ΔW‖², with the range over 3 seeds:

1.5B7B
vanilla LoRA0.689–0.694%0.215–0.219%
rsLoRA / A-shrink / ramp rungs0.671–0.688%0.174–0.191%
full at submitted λ0.677–0.684%0.186–0.191%
anchor at ~1% share0.311–0.319%0.128–0.129%
anchor at ~10% share0.156–0.162%0.066%
random-subspace reference0.608%0.128%

This is the direct answer to "why it works". The acting anchor removes 77% (1.5B) and 70% (7B) of the update's energy from the pretrained principal subspace, and the effect grows monotonically with its gradient share. No other component brings the fraction below the random reference. Vanilla LoRA in fact concentrates its update slightly above the random reference, at 0.69% against 0.61%.

Accuracy consequence. We extended the seeds with counts fixed before the new runs (1.5B to n=9, 7B to n=5). We scored on GSM8K and on held-out SVAMP, which was never used for any choice. Each comparison holds the seed, the data subset and the evaluation precision fixed, and only λ differs:

acting − submittedGSM8KSVAMPper-problem McNemar, both datasets
1.5B, n=9+0.35 ± 0.67+0.51 ± 0.25, 9/9 seedsp = 0.012
7B, n=5+0.33 ± 0.46+0.16 ± 0.17p = 0.076

Fisher's combination across the two scales gives p = 0.007. Against vanilla LoRA, the acting anchor is at statistical parity over both datasets: −0.10 pp at 1.5B and −0.06 pp at 7B, where the submitted configuration was behind. So the anchor's contribution is real once it is delivered.

[Part 3/4] Q2: generalisation beyond mathematical reasoning

We agree that the submission does not answer this question. Additional math sets such as MATH-500 would not answer it either, so we ran a genuinely non-mathematical track:

  • Training data: Alpaca instruction tuning, with the same budget and protocol as the math runs.
  • Evaluation: ARC-Challenge, HellaSwag, BoolQ and Winogrande, scored by length-normalised log-likelihood over the answer options.
  • Comparison arm: vanilla LoRA, matched seed by seed.
  • Reference: the zero-shot base model.

Three seeds, submitted configuration:

ARC-CHellaSwagBoolQWinograndemacro
base, zero-shot37.2964.6073.3365.4360.16
vanilla LoRA38.9464.3374.4866.2561.00
Stable-LoRA⁺39.4564.2274.7366.4861.22

Tuning on Alpaca improves on the base model by +0.84 pp macro for vanilla, so the track is sensitive enough to detect an effect. Paired by seed, Stable-LoRA⁺ is ahead of vanilla by +0.22 ± 0.08 pp (+0.12 / +0.27 / +0.26), positive at 3 of 3 seeds, and it is ahead on 3 of the 4 tasks.

Extended to six seeds, with the anchor acting. λ was set by the same 10%-share rule. Calibrated on Alpaca data, the rule gives 5.08e3, close to the 4.19e3 measured on math data, so it transfers across tasks:

macro, n=6mean− vanillaseeds positive− submittedseeds positive
vanilla LoRA61.10——
submitted61.26+0.15 ± 0.195/6——
anchor acting61.36+0.26 ± 0.205/6+0.11 ± 0.036/6 (p = 0.031)

On ARC-Challenge, the task furthest from the training data, the acting anchor is +0.75 pp above vanilla at 5 of 6 seeds (39.99 vs 39.23). HellaSwag and BoolQ are flat. Making the anchor act improves the transfer result at every seed. This fits the mechanism in Part 2: keeping the update out of the pretrained principal directions matters most when the evaluation task differs from the tuning data.

Scope of the claim. The gains are small, a quarter of a point macro, and below our single-run noise floor of ±0.53 pp. We will therefore state that the method transfers to non-mathematical tasks without harm and with a small, consistent gain, rather than claiming broad generalisation. We also checked forgetting, scoring the math-tuned adapters on the same four tasks. There the acting anchor is +0.14 (1.5B) / +0.03 (7B) relative to vanilla, which is no measurable difference.

Comments on the paper

1 comment

As only a few days remain in the author-reviewer discussion period, I encourage reviewers to review the authors' rebuttal and actively participate in the discussion. In particular, please clarify which concerns have been addressed and identify any remaining concerns or questions that may require further discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.