acceptodds
Withdrawn from ICLR 2027

Same Adapter Function, Different GRPO Dynamics: Factor-Scale Confounding in LoRA Optimization

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Low-Rank Adaptation (LoRA) factorizes each weight update into two trainable matrices, so the same adapter function admits multiple factor scales. Because standard optimizers update these factors separately, functionally identical initializations can receive different updates, confounding the attribution of gains from non-zero LoRA initializations. We study this ambiguity using function-preserving scalar gauges, an identical frozen-rollout control, and reciprocal factor learning rates. Under online Group Relative Policy Optimization (GRPO), shared-rate AdamW amplifies early adapter-product displacement by up to and separates trajectories from identical initial policies, while reciprocal rates substantially reduce this sensitivity across Qwen3 models up to 8B. A teacher-forced SFT control reproduces the local coordinate effect without rollout, reward, or advantage estimation. These results show that factor-scale sensitivity is an optimizer-level confound that is not unique to GRPO, while online policy feedback can make it consequential for early RL trajectories.

Primary area:
optimization
Keywords:
low-rank adaptation, LoRA optimization, reinforcement learning, GRPO, parameter-efficient fine-tuning, reparameterization invariance, optimization geometry

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_VY9dOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how relative factor scaling in LoRA affects training dynamics during online GRPO and SFT. Multiplying one low-rank factor by a positive constant and dividing the other by the same constant leaves the initial adapter weight and policy outputs unchanged, yet scales the two factor gradients in opposite directions. Under AdamW with a shared learning rate, coordinate-wise normalization turns this factor imbalance into skewed updates to the effective adapter weight, causing functionally identical initializations to follow different optimization trajectories. Using exact power-of-two rescalings, frozen-rollout GRPO, and teacher-forced SFT on Qwen3 (1.7B-8B) and Llama-3.2-3B, the authors show that scaling the two factor learning rates inversely with the gauge constant largely restores consistent training dynamics.

Critical strengths
  • The experimental setup is well thought out: using exact power-of-two BF16 rescalings (so initial logits match to machine precision in Table 7) together with frozen-rollout one-step GRPO (Table 6) makes it easy to see that the divergence comes from AdamW's coordinate step rather than rollout noise.
  • I appreciate the candor in Appendices O.1-O.2, where the authors report that native PiSSA is already balanced and none of the 19 real checkpoints they checked follow a global scalar gauge.
Critical weaknesses
  1. Scalar rescaling Tc(B,A)=(cB,A/c)\mathcal{T}_c(B,A)=(cB,A/c) (Eq. 2) is just the 1D subcase R=cIrR=cI_r of the (BR,R−1A)(BR,R^{-1}A) symmetry solved by LoRA-RITE (Yen et al., 2025; Table 19), and Prop. 1 is a direct substitution. Crucially, Apps. O.1-O.2 show native PiSSA is balanced ([0.9998,1.0002][0.9998,1.0002]) and 0/190/19 real adapters fit a global scalar gauge, so Eq. (6) mostly fixes a synthetic perturbation.
  2. In 200-step runs, weight decay is not rescaled with cc. Under AdamW decay θ←(1−ηgλg)θ\theta \leftarrow (1-\eta_g\lambda_g)\theta, setting ηA=η/c,ηB=cη\eta_A=\eta/c, \eta_B=c\eta (Eq. 6) with shared λ=0.01\lambda=0.01 (Table 4) shrinks AcA_c by 1−ηλ/c1-\eta\lambda/c and BcB_c by 1−cηλ1-c\eta\lambda, as Eq. (8) (λA=cλ,λB=λ/c\lambda_A=c\lambda, \lambda_B=\lambda/c) is only used in Table 6. Over 200 steps, this O(cηλ)O(c\eta\lambda) asymmetry confounds Fig. 5(c)'s c>1c>1 undershoot.
  3. The 200-step Qwen3 runs (Figs. 4-5, Table 3) omit the shared-LR baseline, and step-0 accuracy on identical models varies by 3.91 pp in Table 17 (App. D.2), exceeding Table 3's AUC gaps.
Questions for authors
  1. Why was decay rescaling (Eq. 8) omitted in 200-step runs (Table 4), and does (1−cηλ)(1-c\eta\lambda) shrinkage on BcB_c cause Fig. 5(c)'s c>1c>1 undershoot?
  2. Why do shared-LR runs at c=1/8,8c=1/8, 8 beat c=1c=1 at step 10 in Table 17 (69.53% vs. 58.59%), and how do they do at 200 steps?

Above all, my major concern is the synthetic scalar-gauge scope, it's my pleasure to discuss.

Reviewer_j4MbOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies a parametrisation ambiguity in LoRA training. Specifically, in LoRA, we have DeltaW=BADeltaW = BA and the same update can be written using infinitely many equivalent factor pairs, including the scalar transform $(B,A) \rarrow (cB, \frac{A}{c}). This can lead to different optimisation steps, as the optimizer operates separately on A and B.The paper proposes using such iso-functional scalar gauges as a controlled attribution test for studies of non-zero LoRA initialisation.

In their experiments, the authors show that shared-rate AdamW can produce 6x differences between functionally identical factorisations and online GRPO may amplify the differences through future rollouts.

Critical strengths
  • The scalar-gauge construction changes factor co-ordinates while preserving the adapter product and initial policy. The frozen-rollout experiment then holds response tokens, rewards, and advantages fixed, providing a particularly clean demonstration that the immediate divergence originates from factor-wise optimization rather than the online sampling loop.
  • The paper tests the mechanism across GRPO and teacher-forced SFT, multiple model scales and families. The manuscript also appropriately distinguishes reciprocal scalar-rate correction from general transformation-invariant optimisers.
Critical weaknesses
  • The proposed scalar gauge is a restricted and somewhat artificial special case of the full LoRA reparameterization ambiguity and prior work already targets more general cases. For example, LoRA-RITE already formulates the more general invariance problem and explicitly includes scalar scale invariance.
  • The paper frames GRPO prominently even though the SFT experiment demonstrates that the underlying phenomenon is a generic optimiser effect. GRPO potentially amplifies or propagates it, but does not change its nature.
  • The paper reports dramatic product-space differences, yet task-level differences are much smaller and mixed in sign. This raises the possibility that the model is highly non-invariant in parameter space while remaining relatively robust in function space.
Questions for authors
  • The product-space differences are much larger than the downstream task-level differences. How should this gap be interpreted? If final task performance largely converges despite strongly different product-space paths, what practical failure does the method actually prevent?
  • How much extra divergence is specifically caused by the online GRPO feedback loop compared with multi-step fixed-data optimization?
  • How sensitive are the results to learning rate, Adam ϵ\epsilon, weight decay, clipping, rank, and LoRA α\alpha?
  • Can the reciprocal-rate correction be inferred automatically from arbitrary factor geometry, or is it useful only when cc is deliberately imposed and known?
Reviewer_AAMHOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies a typical optimization phenomenon in learning LoRA weights, namely the invariance, (BA=cBAc−1BA = c B Ac^{-1}) under which the scales of A and B will be very different. The paper attributes this phenomenon to the optimizer using a shared learning rate.

Critical strengths

The paper identifies the phenomenon of "displacement" in LoRA optimization.

Critical weaknesses

It is unclear what problem this paper is actually trying to solve.

Questions for authors

My biggest concern is the motivation. The "displacement" of the optimization trajectory is a common phenomenon. What negative effects does it cause? Why should we care about this phenomenon and seek to resolve it?

Minor Comments: Scalars and vectors use the same notation, which is confusing. Some symbols are never defined, e.g., is ll the layer index?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.