acceptodds
Under review as a conference paper at ICLR 2027

RANK IS NOT A TRUST REGION: OPTIMIZER-INDUCED GEOMETRY IN LORA REINFORCEMENT LEARNIN

Abstract

Low-rank adaptation (LoRA) is widely used for reinforcement learning with verifiable rewards (RLVR), and a low rank is commonly expected to keep the policy close to where it started. We show that rank is not a trust region: in GRPO training of Qwen3.5-4B, rank-4 and rank-64 adapters that update the weights equally far differ by in on-policy KL, and by under an explicit KL penalty. Two updates matched in rank, norm, and row and column subspaces differ by in held-out KL. The reason is that policy KL depends on the direction of an update and not only on its size. Under standard initialization, the down-projection of LoRA barely moves, so its rows act as nearly fixed random probes of the gradient. As the gradient concentrates in directions to which the policy is most sensitive, a higher rank supplies more probes that capture this sensitivity, so the KL cost of a matched displacement grows with rank. Changing the probes tests this mechanism: probes aligned with the gradient reverse the rank ordering of curvature and, when KL stays local, reverse the rank ordering of KL as well. Directional Fisher sensitivity, which measures curvature along the realized update per unit norm, predicts KL on held-out sequences in two model families. Policy movement also does not determine whether an update helps, because KL is locally even in the update direction while utility change is odd. Stability in LoRA-based RLVR should therefore be monitored and constrained in distribution space, with rank chosen for capacity rather than as a trust-region parameter.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Low-Rank Adaptation (LoRA), Reinforcement Learning with Verifiable Rewards (RLVR), Parameter-Efficient Fine-Tuning (PEFT), Trust Region, Policy KL Divergence, Fisher Information, Large Language Models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ufpyOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper argues that LoRA rank does not act as a policy-space trust region in RLVR. Using rank-stabilized LoRA during GRPO training of Qwen3.5-4B on GSM8K, the authors observe an approximately 11x gap in on-policy KL between rank-64 and rank-4 adapters at matched effective-weight displacement. The separation persists on held-out prompts, under an explicit KL penalty, and on Qwen2.5-1.5B-Instruct. They also construct two updates with identical rank, norm, row space, and column space that differ by 240x in held-out KL. The proposed explanation is a random-probe mechanism: under standard initialization, A barely moves, so its rows act as nearly fixed probes through which the optimizer transforms the gradient. Under the proposed model, higher rank supplies more probes that capture high-curvature gradient components, increasing the KL cost per unit displacement. The mechanism is tested through single-module interventions, and gradient-aligned probes reverse the average rank ordering of KL in online training at a lower learning rate where KL remains local (11.3x with random probes versus 0.58x with aligned probes). Directional Fisher sensitivity predicts held-out KL ratios, with out-of-fold Q² of 0.82 versus 0.73 for rank alone on Granite, and additional replication on Qwen3.5. Finally, the paper distinguishes policy movement from utility: KL is even in the update direction to second order, while first-order utility change is odd, so bounding KL alone does not determine whether an update helps.

Critical strengths
  1. The probe mechanism is tested through targeted interventions, not just correlations. Freezing A, aligning its initialization, and changing its learning rate generally produce the predicted changes. The average reversal of the online KL ordering with aligned probes in the local-KL regime is particularly compelling and is my main reason for leaning toward acceptance.

  2. The practical message is useful for RLVR practitioners who treat rank or weight displacement as a stability proxy. The recommendations—monitor and constrain movement in distribution space, and choose rank for capacity—follow from the controlled comparisons. Directional Fisher sensitivity estimated from a small number of sequences also offers a useful diagnostic for held-out KL.

Critical weaknesses
  1. The online evidence is still limited in duration and coverage. The main multi-seed experiments run for 30 rollout rounds, while the 200-round comparison contains only one trajectory pair. The standard-LoRA controls are useful, but use alpha equal to rank. Longer multi-seed runs and controls holding alpha fixed would clarify how broadly the observed rank effect applies beyond the matched-displacement setting.

  2. The theory is more convincing qualitatively than quantitatively. The random-probe scaling law in Proposition 3 is derived under SGD and a row-side Fisher model, whereas the main experiments use AdamW and full sequence-Fisher measurements. The paper acknowledges the resulting discrepancies, but the conditions under which the approximation remains reliable need clarification. The Fisher predictor also needs uncertainty analysis across modules and evaluation batches, despite the encouraging Qwen3.5 replication.

Questions for authors
  1. The standard-LoRA controls use alpha equal to rank. Does the rank effect persist when alpha is held fixed across ranks, using the same learning rate and training schedule?

  2. Across learning rates and training stages, when does aligned initialization reverse the KL ordering? Please report absolute KL and task performance alongside the rank ratios.

  3. How stable are the predictive gains of Fisher sensitivity, alone and combined with rank, across new seeds and evaluation batches? Can you provide uncertainty or module-level sensitivity analyses?

  4. Appendix C.13 mentions longer generation budgets. For the 200-round rank-4/rank-256 pair, can you report accuracy and truncation at each budget, together with the completion-versus-conditional decomposition in Eq. (60)?

4 comments
AuthorsOpenReview ↗

Response to Reviewer ufpy (1/2)

(1/2) Thank you for the constructive review and for engaging with the probe interventions. We ran new experiments for each of the four questions.

1. Fixed alpha

The matched-displacement runs in Section 3 use α=8\alpha=8 at both ranks with rank-stabilized scaling (KL gap 11.2 ± 2.3× at norm ratio 0.99); standard LoRA with α=r\alpha=r is used for the orientation, MATH and long-horizon pairs. We added standard LoRA with α=8\alpha=8 at both ranks, everything else as in the paper.

Seed∥ΔW∥F\lVert\Delta W\rVert_F r4 / r64on-policy KL r4 / r64KL ratioμˉK\bar\mu_K ratio
00.136 / 0.0340.0118 / 0.01361.1618.5
10.132 / 0.0320.0157 / 0.00710.457.5
20.141 / 0.0330.0495 / 0.00940.193.5

Round 30. μˉK\bar\mu_K is KL per unit squared displacement (on-policy; fixed-prefix gives 20.9, 7.5, 3.7).

With α\alpha fixed, rank 64 moves only a quarter as far as rank 4. Per unit of squared displacement it still costs 3.5–18.5× more KL, so the rank effect is there. The two factors multiply: in seed 0, 0.252×18.5≈1.160.25^2 \times 18.5 \approx 1.16, the measured KL ratio. Total KL therefore no longer follows rank: rank 64 ends slightly above rank 4 in one seed and well below it in the other two. In this common setting a lower rank does not keep the policy closer. Rewards are similar (0.81–1.00). This is in Appendix C.2.

2. When aligned initialization reverses the KL ordering, with absolute KL and task performance

lrA0A_0KL ratio 64/4 at rounds 10 / 20 / 30 (seeds 0, 1, 2)rank-4 KL, round 30GSM8K r4 / r64 (%)
2e-6aligned0.85, 0.40, 0.60 / 0.92, 0.14, 0.44 / 0.996, 0.32, 0.430.15–0.3983.1–84.5 / 82.7–83.3
2e-6random8.9, 10.8, 19.9 / 7.4, 10.0, 15.9 / 8.4, 10.1, 15.45–6e-482.3–83.1 / 82.7–83.6
2e-5aligned1.38, 2.05, 0.67 / 1.07, 1.42, 0.86 / 1.10, 1.67, 0.846.0–12.687.3–91.3 / 87.3–90.7
2e-5random8.6, 17.8, 13.9 / 9.0, 13.9, 12.7 / 9.3, 13.8, 10.70.05–0.1182.9–83.6 / 85.6–86.1

KL in nats per sequence at matched displacement. GSM8K: full test set, 512 tokens, greedy, round 30, range over seeds; all rows use the same evaluation script.

At 2e-6 the ratio is below one at every round in every seed, from 0.14 to 0.996. The reversal comes from rank 4: aligned probes raise its KL by two to three orders of magnitude over random probes, more than they raise the KL of rank 64. At the default rate the aligned runs reach 6–13 nats, the KL ratios move toward one, and only the Fisher-sensitivity ratio (0.22–0.78) is still below one in all nine cases. Task accuracy does not suffer. At 2e-6, aligned and random probes score the same. At 2e-5 the aligned runs move the policy one to two orders of magnitude further than the random runs and score higher, with fewer truncated answers.

AuthorsOpenReview ↗

Response to Reviewer ufpy (2/2)

(2/2)### 3. Stability of the Fisher predictor

We bootstrapped over modules (2000 resamples, keeping the three ranks of a module together and refitting every time) and, separately, over evaluation sequences. The table gives the gain in out-of-fold Q2Q^2 from adding GFG_F to rank.

Targetmodule bootstrap [95%]share > 0sequence bootstrap [95%]
Qwen3.5, 8 modules0.274 [0.127, 0.487]99.7%0.249 [0.199, 0.294]
Granite, sequences 9–160.197 [−0.022, 0.611]96.5%0.204 [0.146, 0.245]
Granite, sequences 65–800.139 [−0.072, 0.432]92.5%0.142 [0.111, 0.179]

On Qwen3.5 the gain is stable either way. On Granite it is stable over sequences, but the module interval reaches zero. Most of that spread comes from one module, the layer-20 output projection; without it the gain is 0.098 and 0.083. The sequences used to estimate the Fisher were not resampled, since per-sequence scores were not stored.

4. Generation budgets for the 200-round pair

Seed 0, as in the paper (rank 256 minus rank 4, round 200):

Benchmark (tokens)Acc r4 / r256truncated r4 / r256ΔAccconditionalcompletioncommon-support Δ [95% CI]
GSM8K (512)85.4 / 92.613.3 / 1.1+7.20−2.15+9.66−0.26 [−1.41, +0.88]
GSM8K (1024)92.1 / 93.12.2 / 0.4+0.99−0.57+1.64+0.31 [−0.93, +1.48]
MATH-500 (1024)63.8 / 72.837.0 / 23.4+9.00−2.97+12.12−1.62 (n = 309)
MATH-500 (2048)81.0 / 83.817.0 / 11.2+2.80−2.92+5.21−2.49 [−4.98, 0.00]

Channels follow Eq. (60) of the submission (Eq. (59) in the revision); the small truncated-bucket term is omitted.

On GSM8K the 512-token gap is a truncation effect and disappears at 1024 tokens. On MATH-500, rank 256 is about 2.5 points worse on problems both ranks finish, at the edge of significance. Section 5.5 reports GSM8K, and Appendix C.13 reports both tasks.

We then ran two more seeds and evaluated all six checkpoints with one script:

Seedcommon-support Δ, 512 tokenscommon-support Δ, 1024 tokens
0+0.44 [−0.70, +1.57]+0.31 [−0.86, +1.40]
1−4.53 [−6.08, −2.89]−6.08 [−7.87, −4.38]
2+0.00 [−1.14, +1.14]−0.08 [−1.24, +1.09]

For seed 0 this script gives +0.44 against −0.26 in the paper. The two evaluations differ only in generation batch size (8 against 4); with left padding in bf16 this changes about half of the greedy rank-4 responses and the correctness of 53 of 1319, yielding a 0.7-point difference between the two evaluation configurations.

Seed 2 behaves like seed 0. Seed 1 does not: after round 90 the entropy of rank 256 climbed from 0.8 to 8.7 nats per token, the training reward dropped to 0.1–0.4 and never recovered, and accuracy fell by 4.5–6.1 points. A first attempt of seed 0, stopped at round 129 by an out-of-memory error, had shown the same rise. Rank 4 was stable in all seeds.

Is the instability caused by rank?

With α=r\alpha=r, rank 256 also moves 7–15× further in weight space than rank 4, so these runs cannot separate rank from step size. We ran two more conditions, each compared with rank 4 of the same seed (Table 19):

ConditionKL r4 / r256common-support Δ, 512common-support Δ, 1024
α=32\alpha=32 (matched displacement), seed 00.30 / 10.1−0.17 [−1.22, +0.78]−0.55 [−1.64, +0.55]
α=32\alpha=32, seed 10.56 / 8.9+0.43 [−0.43, +1.21]−0.16 [−1.17, +0.70]
α=32\alpha=32, seed 20.29 / 11.2+0.09 [−0.96, +1.13]−0.70 [−1.78, +0.47]
α=256\alpha=256 + KL penalty 0.04, seed 10.56 / 18.2+0.00 [−1.04, +1.04]+0.08 [−1.01, +1.09]

At matched displacement, rank 256 trained stably in all three seeds, including seed 1, and still ended with 16–38× the KL of rank 4, with no detectable loss in conditional accuracy. Adding a KL penalty also kept seed 1 stable (KL 18 instead of 438). In our runs the instability required both a large step and no KL penalty. Section 5.5 now leads with the matched-displacement runs and reports the instability.

On the quantitative accuracy of the theory

See our answer to Reviewer 2, Q2(b). On the same statistic, the prediction from the actual A0A_0 is within 1.2× of each measurement, and the measured ratios are 10–23% below the random-probe ratio of expectations, with the ordering preserved in all 200 draws (Appendix C.7).

Thank you for the thorough additional experiments, which address my questions on fixed α, the conditions under which aligned probes reverse the KL ordering, the stability of the Fisher predictor, and the generation budgets for the long-horizon runs, and I will maintain my positive score.

Thank you for reading the additional results and for the questions that led to them. The fixed-α runs and the matched-displacement 200-round runs in particular sharpened the paper's claim, and both are in the revision (Appendix C.2 and C.13, Section 5.5).

Reviewer_dE9LOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors compare LoRA updates whose rank, norm, and subspaces have been matched, and report large differences in policy KL. Their account focuses on the directions selected by the optimizer and their Fisher sensitivity. Nearly fixed random probes are proposed to explain the rank dependence. The evidence includes interventions on these probes, a reversal of the local KL ordering, and predictions of KL ratios on held-out data.

Critical strengths
  1. The controlled comparisons provide concrete evidence that rank and displacement do not determine realized policy KL. The displacement-matched experiments account for step energy and path coherence (Section 3, p. 4, lines 201–215), while the orientation experiment preserves rank, norm, and row and column spaces (p. 5, lines 216–225). Independent held-out evaluation of the selected orientation pair strengthens the latter result (Appendix C.11, p. 28, lines 1490–1501).

  2. Interventions on the initial probes test the proposed mechanism by changing the rank dependence (Section 5.2, Table 1, p. 8), including an online KL-ordering reversal on average under gradient-aligned initialization when KL remains local (Section 5.4, p. 9, lines 434–440). The held-out KL-ratio predictions further show that directional Fisher sensitivity contains information beyond rank alone (Section 5.3, p. 8, lines 420–431).

Critical weaknesses
  • W1. The introduction connects learning–forgetting evidence to proximity to initialization and an implicit trust region (Section 1, p. 1, lines 38–48). Biderman et al. (2024, Sections 4.2–4.3 and 4.5 of that paper) support this motivation by reporting rank-dependent forgetting and discussing the possibility of replacing a KL penalty. However, such evidence does not establish rank-monotonic reference-policy KL or a trust-region guarantee. The manuscript defines the trust-region interpretation, but I would like to see a more precise link between the cited evidence and the claimed use of rank as a KL-control proxy (Q1; see also Appendix A, p. 14). This affects how I assess the manuscript's contribution beyond the directional KL differences it demonstrates.

  • W2. The connection between Proposition 3(1) and the measured rank dependence needs clarification. Under SGD, the proposition predicts a rank ordering that is reversed in the step-30 SGD results for q and up in Table 10 (p. 26). The theory uses a row-side Fisher model, whereas Table 10 uses the full sequence Fisher. What explains this reversal, and how the theory and the Adam-style predictions in Appendix C.7 support the reported agreement with AdamW, remains unclear to me (Q2(a); Section 5.1, p. 7, lines 355–359).

    The quantitative comparison also uses different statistics: the theoretical ratio of 8.7 is based on E[qr]/E[er]\mathbb{E}[q_r]/\mathbb{E}[e_r], while the measured ratios of 7.0–8.2 use individual normalized sensitivities qr/erq_r/e_r. Appendix C.6 acknowledges variation across draws and an approximation for sign-normalized updates, but does not establish the accuracy of this comparison in the reported setting (Q2(b); p. 24, Eq. (59), lines 1277–1279).

Questions for authors
  • Q1 (W1). Which prior findings or practices motivate treating rank as a control on reference-policy KL (Section 1, p. 1, lines 38–48)? Please distinguish empirical regularization, rank-monotonic KL, and a trust-region guarantee, and specify which claims your results address.

  • Q2(a) (W2). Which assumptions or measurement differences account for the reversed SGD ordering in Table 10 (p. 26)? How do the predictions for Adam-style updates in Appendix C.7 (p. 25, Tables 8–9) support carrying over Proposition 3(1)'s rank scaling to AdamW?

  • Q2(b) (W2). In Section 5.1 (p. 7, lines 355–359), the theoretical and measured rank ratios use different statistics. What in the existing analysis or measurements supports treating them as comparable? Could this difference substantially change the comparison between ranks?

3 comments
AuthorsOpenReview ↗

Response to Reviewer dE9L

Thank you for the careful questions. Both led to new analyses, which are now in the appendix.

W1 / Q1. Which prior findings motivate treating rank as a control on reference-policy KL

The motivating evidence is of the first kind: empirical regularization. Biderman et al. (2024) show that LoRA forgets less than full fine-tuning and that forgetting depends on rank. Sun et al. (2023) report that removing the KL penalty did not hurt RLHF with LoRA. Neither measures policy KL, and neither claims a trust-region guarantee. Our results speak to the other two levels. Under the default initialization KL does grow with rank at matched displacement (11×), but aligned probes reverse this, so the monotonicity comes from the initialization. And no guarantee follows from rank, norm or subspace, since two updates matched in all three differ by 240× in held-out KL. The introduction now draws this distinction.

W2 / Q2(a). The reversed SGD ordering in Table 10

Part 1 of Proposition 3 in the submission (Proposition 2 in the revision) assumes that the probes stay fixed. To test whether the SGD runs violate it, we reran SGD on Granite layer 20 with the same seed, batch and step-norm matching, saving the factors, and added a run with AA frozen.

ModuleAAμL(4)\mu_L(4)μL(64)\mu_L(64)ratio 64/4drift of AA (r4 / r64)
uptrained1.9510.7780.408.4% / 3.0%
upfrozen0.2410.9854.09–
qtrained16.5010.950.66– / 2.0%
qfrozen2.1807.9753.66–

Step-30 Fisher sensitivity of the LoRA arm (trained rows as in Table 10); drift is relative to ∥A0∥F\lVert A_0\rVert_F.

To match the AdamW step norm, SGD needs a learning rate of 0.84 on up (0.41–0.42 on qq). At that rate, on a fixed batch, AA moves toward the top gradient directions. The move is small, 8.4% of the norm at rank 4, but it is aimed at high curvature, and the rank-4 sensitivity ends up about eight times higher than with AA frozen. With AA frozen, the ordering of Part 1 holds on both modules. So the reversal comes from directed drift of AA, the same effect as a 20× learning rate for AA on a fixed batch.

For AdamW, the row-side gradient second moment is close to proportional to the Fisher (median matrix cosine 0.96), and predictions from the actual gradient and A0A_0 match the measured ratios (Appendix C.7).

Q2(b). Different statistics in Section 5.1

You are right. The 8.7 was a per-module ratio of expectations, and the measured 7.0–8.2 are single draws for the whole model. We now compare like with like.

Projecting the actual gradients onto the actual A0A_0 predicts 7.1, 6.0 and 7.6 for the three seeds. The measured values are 7.5, 7.0 and 8.2. To see how much a single draw can vary, we sampled random probes for ranks 4 and 64 from the row-side Fisher spectra of all 40 adapted Qwen3.5 modules, 200 times:

Update rulemedian95% rangeratio of expectations
SGD10.45.1–18.89.6
sign-normalized, 60 steps9.36.6–13.19.1

The prediction from the actual A0A_0 is within 1.2× of each measurement. The measured ratios are 10–23% below the sign-normalized ratio of expectations (9.1) and sit in the lower quarter of its range. None of the 200 draws falls below 5. Section 5.1 and Appendix C.7 are revised accordingly.

Thank you for the response and the extra analyses. The SGD experiment with A frozen and the revised comparison of the statistics largely resolve W2 for me.

For W1, I was not convinced by the evidence that practitioners use rank to control policy KL, or by the importance of the new findings from testing that premise. I see value in the initialization experiments, but my assessment of the contribution is unchanged. so I keep my score.

Thank you for the follow-up, and for confirming that W2 is resolved.

On W1, we agree that no prior work uses rank as an explicit control on policy KL; the evidence we cite measures forgetting or task loss. Our paper answer a different question: In KL-free LoRA RLVR, the adapter is the only structural limit on how far the policy moves, and its rank, scaling and initialization are chosen for capacity, convergence or forgetting, not for KL. The paper shows what these choices do to policy KL:

  • Rank sets the KL cost of a given weight displacement through the number of random probes; it does not bound KL. With α\alpha fixed in standard LoRA, rank 4 moves the weights four times farther than rank 64 and ends with higher KL in two of three seeds.
  • Initialization decides the sign of the rank effect. With gradient-aligned A0A_0, the idea behind gradient-based initializations such as LoRA-GA, increasing the rank lowers KL at matched displacement, at a learning rate where KL stays local; with the default initialization it raises KL by about 11×.
  • In 200-round training, the instability we observed came with large displacement and no KL penalty: rank 256 at matched displacement, or with a KL penalty, trained stably.

These results describe how standard LoRA design choices act on policy movement, independent of whether rank is viewed as a trust region. We will revise the introduction to lead with this question and present the trust-region reading as one motivation rather than as the premise.

Reviewer_NicbOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The authors claim that rank is thought of as a way to keep a network close to its pretraining outputs during finetuning. They describe KL as a measurement of this idea, and demonstrate that in some ways rank/weight-delta is not a perfect proxy for KL. Specifically, KL changes much more for some initializations of LoRA than others. Larger rank increases the number of directions that LoRA can impact the weights, and therefore the probability that a LoRA direction is aligned with a direction of rapid policy-variation. A series of experiments demonstrate that the same weight-displacement leads to more policy impact in high-rank LoRA. In the early-training regime the authors successfully attribute this to the alignment of the LoRA projection to high-policy-curvature directions of weight updates — by selecting the LoRA down-projection appropriately, a significant amount of policy-modification can be accomplished with only small weight change at low LoRA dimension. The authors formalize this sensitivity in terms of the Fisher matrix, relate this idea to trust-regions found elsewhere in RL finetuning, and argue that constraining rank or weight-movement is insufficient to regularize a fine-tuned policy.

Critical strengths

I thought parts of the paper were interesting and informative. I thought that the discussion relating to down-projection stationarity was a helpful framing. The experiments contrasting default vs “aligned” down-projections were informative, and suggest possibly helpful methodology in choosing LoRA initializations. The question being asked is a small but reasonable one, however as I describe below I think the presentation is very flawed.

Critical weaknesses

I think that this paper is a relatively careful investigation into a claim, but has significant problems. First, I think there’s a contradiction within the first two sentences of the abstract: “low rank is expected to keep policy close to start“, followed by a sentence demonstrating that KL divergence scales roughly with rank. The next claim in the abstract — that policy modification depends on gradient direction — is not under dispute by anyone. Similarly, that KL is even while utility is odd — that’s a consequence of KL being minimized at initialization, and too obvious to have a place in the abstract, and definitely does not need whole subsection. This is true of Figure 1D and surrounding discussion — I don’t think the discussion of trust regions is very cogent, and 1D doesn’t add anything. I felt that this critique applies in a number of places in this paper — that math, jargon, or results were used in a way that made it difficult to understand the central claim, rather than to clarify it.

Besides presentation, there’s at least one methodological issue I noticed. The experiments are all done on a very small number of update-steps, which seems like it would have a big impact on the stationarity of A. As presented it’s unclear the utility of these results to standard training regimes because the conclusions don’t necessarily hold if A changes significantly during training. However, the experiments are generally reasonable.

The authors make clear the central role of LLMs in the paper's writing and construction. I think the presentational difficulties stem from this. Too many things are presented as novel contributions that aren't, and the signal gets lost in the noise. I think much of the math in this paper is more confusing than helpful, or more formalized than it needs to be for the intuition it provides (e.g. proposition 1 and 4 and 6). The paper would be vastly improved by simplifying the presentation and focusing it on one clear idea, with one clear takeaway.

Questions for authors

I am curious whether the results regarding initializations hold throughout training, or whether this is a description of the early-training regime. I don't think that's central to my review thoguh.

1 comment
AuthorsOpenReview ↗

Response to Reviewer Nicb

We address the points in order. The changes are in the abstract, the introduction, and Section 4, and there are new runs on the stationarity of AA.

1. The abstract appears to contradict itself

Under the default initialization the low-rank intuition matches our data: at matched displacement rank 64 moves the policy about 11× more than rank 4. The ordering, however, comes from the initialization. When we initialize A0A_0 along the leading gradient directions, rank 64 moves the policy less than rank 4: at a learning rate where KL stays local, the KL ratio is below one at rounds 10, 20 and 30 in all three seeds, against 7–20 with random A0A_0 (Section 5.4). And at a single rank, two updates with the same norm and the same row and column spaces differ by 240× in held-out KL (Section 3). The revised abstract states this claim in its second sentence: rank changes how far the policy moves but does not control it.

2. Direction dependence and the even/odd statement are not new

We agree that both are standard properties of KL, and the revision presents them that way. The directional radius is now a sentence in Section 4.1 rather than a proposition, and Section 4.3 is shorter. Our contribution is the specific way rank enters for LoRA. Under the standard initialization the rows of A0A_0 barely move and act as random probes of the gradient, and rank is the number of probes. Changing the probes changes the rank effect in the direction the model predicts: freezing AA strengthens it, aligning A0A_0 with the gradient reverses it, and a larger learning rate for AA weakens it on fixed batches.

Section 5.5 uses the parity statement to separate policy movement from utility. For example, aligned probes at the default learning rate reach 6–13 nats of on-policy KL per sequence, yet score 87–91% on GSM8K, above the 83–86% of random probes at the same learning rate (Table 14).

3. Few update steps and the stationarity of AA

After 200 online rounds, AA has moved by at most 6.7% of its initial norm in all six runs of the 200-round pair, and its row space overlaps the initial one at 0.989 or more (Appendix C.6). Online, successive updates of AA are nearly uncorrelated (cosine 0.16–0.46), so the drift does not accumulate in one direction.

The rank effect also survives longer training. We trained rank 256 with α=32\alpha=32, which gives it the same weight displacement as rank 4, for 200 rounds in three seeds. At round 200 its on-policy KL is 16–38× that of rank 4 (Appendix C.13).

Drift does matter when it has a consistent direction. Under SGD on a fixed batch, AA turns toward the leading gradient directions, and this is where Table 10 shows a reversed ordering. With AA frozen and everything else unchanged, the ordering comes back: the rank-64/rank-4 ratio goes from 0.40 to 4.09 on up and from 0.66 to 3.66 on qq (Table 11). Section 4.2 discusses this case after Part 3 of Proposition 2 (Proposition 3 in the submission), which bounds how far a given drift moves the probes.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of revisions

We have uploaded a revision; changes are in blue. In short:

  • The abstract and introduction state the claim directly. Rank changes how far the policy moves, but it does not control it: a low rank keeps the policy closer only under the default initialization. The introduction notes that prior work treating rank as a stability knob measures forgetting or task loss, not policy KL.
  • Section 5.1 compares the probe model with measurement on the same statistic, and Appendix C.7 shows how much a single draw of A0A_0 varies. The SGD reversal in Table 10 goes away when AA is frozen (Appendix C.8).
  • New runs: standard LoRA with α\alpha fixed across ranks (C.2), GSM8K for all aligned and random runs (C.9), bootstraps for the Fisher predictor (C.4), and three seeds of the 200-round pair with two extra conditions (C.13).
  • In the 200-round pair with α=r\alpha=r, rank 256 became unstable in one of three seeds. At matched displacement, or with a KL penalty, it stayed stable while reaching 16–38× the KL of rank 4. Section 5.5 reports both.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.