- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper argues that LoRA rank does not act as a policy-space trust region in RLVR. Using rank-stabilized LoRA during GRPO training of Qwen3.5-4B on GSM8K, the authors observe an approximately 11x gap in on-policy KL between rank-64 and rank-4 adapters at matched effective-weight displacement. The separation persists on held-out prompts, under an explicit KL penalty, and on Qwen2.5-1.5B-Instruct. They also construct two updates with identical rank, norm, row space, and column space that differ by 240x in held-out KL. The proposed explanation is a random-probe mechanism: under standard initialization, A barely moves, so its rows act as nearly fixed probes through which the optimizer transforms the gradient. Under the proposed model, higher rank supplies more probes that capture high-curvature gradient components, increasing the KL cost per unit displacement. The mechanism is tested through single-module interventions, and gradient-aligned probes reverse the average rank ordering of KL in online training at a lower learning rate where KL remains local (11.3x with random probes versus 0.58x with aligned probes). Directional Fisher sensitivity predicts held-out KL ratios, with out-of-fold Q² of 0.82 versus 0.73 for rank alone on Granite, and additional replication on Qwen3.5. Finally, the paper distinguishes policy movement from utility: KL is even in the update direction to second order, while first-order utility change is odd, so bounding KL alone does not determine whether an update helps.
Critical strengths
-
The probe mechanism is tested through targeted interventions, not just correlations. Freezing A, aligning its initialization, and changing its learning rate generally produce the predicted changes. The average reversal of the online KL ordering with aligned probes in the local-KL regime is particularly compelling and is my main reason for leaning toward acceptance.
-
The practical message is useful for RLVR practitioners who treat rank or weight displacement as a stability proxy. The recommendations—monitor and constrain movement in distribution space, and choose rank for capacity—follow from the controlled comparisons. Directional Fisher sensitivity estimated from a small number of sequences also offers a useful diagnostic for held-out KL.
Critical weaknesses
-
The online evidence is still limited in duration and coverage. The main multi-seed experiments run for 30 rollout rounds, while the 200-round comparison contains only one trajectory pair. The standard-LoRA controls are useful, but use alpha equal to rank. Longer multi-seed runs and controls holding alpha fixed would clarify how broadly the observed rank effect applies beyond the matched-displacement setting.
-
The theory is more convincing qualitatively than quantitatively. The random-probe scaling law in Proposition 3 is derived under SGD and a row-side Fisher model, whereas the main experiments use AdamW and full sequence-Fisher measurements. The paper acknowledges the resulting discrepancies, but the conditions under which the approximation remains reliable need clarification. The Fisher predictor also needs uncertainty analysis across modules and evaluation batches, despite the encouraging Qwen3.5 replication.
Questions for authors
-
The standard-LoRA controls use alpha equal to rank. Does the rank effect persist when alpha is held fixed across ranks, using the same learning rate and training schedule?
-
Across learning rates and training stages, when does aligned initialization reverse the KL ordering? Please report absolute KL and task performance alongside the rank ratios.
-
How stable are the predictive gains of Fisher sensitivity, alone and combined with rank, across new seeds and evaluation batches? Can you provide uncertainty or module-level sensitivity analyses?
-
Appendix C.13 mentions longer generation budgets. For the 200-round rank-4/rank-256 pair, can you report accuracy and truncation at each budget, together with the completion-versus-conditional decomposition in Eq. (60)?
4 comments
Response to Reviewer ufpy (2/2)
(2/2)### 3. Stability of the Fisher predictor
We bootstrapped over modules (2000 resamples, keeping the three ranks of a module together and refitting every time) and, separately, over evaluation sequences. The table gives the gain in out-of-fold from adding to rank.
| Target | module bootstrap [95%] | share > 0 | sequence bootstrap [95%] |
|---|---|---|---|
| Qwen3.5, 8 modules | 0.274 [0.127, 0.487] | 99.7% | 0.249 [0.199, 0.294] |
| Granite, sequences 9–16 | 0.197 [−0.022, 0.611] | 96.5% | 0.204 [0.146, 0.245] |
| Granite, sequences 65–80 | 0.139 [−0.072, 0.432] | 92.5% | 0.142 [0.111, 0.179] |
On Qwen3.5 the gain is stable either way. On Granite it is stable over sequences, but the module interval reaches zero. Most of that spread comes from one module, the layer-20 output projection; without it the gain is 0.098 and 0.083. The sequences used to estimate the Fisher were not resampled, since per-sequence scores were not stored.
4. Generation budgets for the 200-round pair
Seed 0, as in the paper (rank 256 minus rank 4, round 200):
| Benchmark (tokens) | Acc r4 / r256 | truncated r4 / r256 | ΔAcc | conditional | completion | common-support Δ [95% CI] |
|---|---|---|---|---|---|---|
| GSM8K (512) | 85.4 / 92.6 | 13.3 / 1.1 | +7.20 | −2.15 | +9.66 | −0.26 [−1.41, +0.88] |
| GSM8K (1024) | 92.1 / 93.1 | 2.2 / 0.4 | +0.99 | −0.57 | +1.64 | +0.31 [−0.93, +1.48] |
| MATH-500 (1024) | 63.8 / 72.8 | 37.0 / 23.4 | +9.00 | −2.97 | +12.12 | −1.62 (n = 309) |
| MATH-500 (2048) | 81.0 / 83.8 | 17.0 / 11.2 | +2.80 | −2.92 | +5.21 | −2.49 [−4.98, 0.00] |
Channels follow Eq. (60) of the submission (Eq. (59) in the revision); the small truncated-bucket term is omitted.
On GSM8K the 512-token gap is a truncation effect and disappears at 1024 tokens. On MATH-500, rank 256 is about 2.5 points worse on problems both ranks finish, at the edge of significance. Section 5.5 reports GSM8K, and Appendix C.13 reports both tasks.
We then ran two more seeds and evaluated all six checkpoints with one script:
| Seed | common-support Δ, 512 tokens | common-support Δ, 1024 tokens |
|---|---|---|
| 0 | +0.44 [−0.70, +1.57] | +0.31 [−0.86, +1.40] |
| 1 | −4.53 [−6.08, −2.89] | −6.08 [−7.87, −4.38] |
| 2 | +0.00 [−1.14, +1.14] | −0.08 [−1.24, +1.09] |
For seed 0 this script gives +0.44 against −0.26 in the paper. The two evaluations differ only in generation batch size (8 against 4); with left padding in bf16 this changes about half of the greedy rank-4 responses and the correctness of 53 of 1319, yielding a 0.7-point difference between the two evaluation configurations.
Seed 2 behaves like seed 0. Seed 1 does not: after round 90 the entropy of rank 256 climbed from 0.8 to 8.7 nats per token, the training reward dropped to 0.1–0.4 and never recovered, and accuracy fell by 4.5–6.1 points. A first attempt of seed 0, stopped at round 129 by an out-of-memory error, had shown the same rise. Rank 4 was stable in all seeds.
Is the instability caused by rank?
With , rank 256 also moves 7–15× further in weight space than rank 4, so these runs cannot separate rank from step size. We ran two more conditions, each compared with rank 4 of the same seed (Table 19):
| Condition | KL r4 / r256 | common-support Δ, 512 | common-support Δ, 1024 |
|---|---|---|---|
| (matched displacement), seed 0 | 0.30 / 10.1 | −0.17 [−1.22, +0.78] | −0.55 [−1.64, +0.55] |
| , seed 1 | 0.56 / 8.9 | +0.43 [−0.43, +1.21] | −0.16 [−1.17, +0.70] |
| , seed 2 | 0.29 / 11.2 | +0.09 [−0.96, +1.13] | −0.70 [−1.78, +0.47] |
| + KL penalty 0.04, seed 1 | 0.56 / 18.2 | +0.00 [−1.04, +1.04] | +0.08 [−1.01, +1.09] |
At matched displacement, rank 256 trained stably in all three seeds, including seed 1, and still ended with 16–38× the KL of rank 4, with no detectable loss in conditional accuracy. Adding a KL penalty also kept seed 1 stable (KL 18 instead of 438). In our runs the instability required both a large step and no KL penalty. Section 5.5 now leads with the matched-displacement runs and reports the instability.
On the quantitative accuracy of the theory
See our answer to Reviewer 2, Q2(b). On the same statistic, the prediction from the actual is within 1.2× of each measurement, and the measured ratios are 10–23% below the random-probe ratio of expectations, with the ordering preserved in all 200 draws (Appendix C.7).
Thank you for the thorough additional experiments, which address my questions on fixed α, the conditions under which aligned probes reverse the KL ordering, the stability of the Fisher predictor, and the generation budgets for the long-horizon runs, and I will maintain my positive score.
Thank you for reading the additional results and for the questions that led to them. The fixed-α runs and the matched-displacement 200-round runs in particular sharpened the paper's claim, and both are in the revision (Appendix C.2 and C.13, Section 5.5).
Response to Reviewer ufpy (1/2)
(1/2) Thank you for the constructive review and for engaging with the probe interventions. We ran new experiments for each of the four questions.
1. Fixed alpha
The matched-displacement runs in Section 3 use α=8 at both ranks with rank-stabilized scaling (KL gap 11.2 ± 2.3× at norm ratio 0.99); standard LoRA with α=r is used for the orientation, MATH and long-horizon pairs. We added standard LoRA with α=8 at both ranks, everything else as in the paper.
Round 30. μˉK is KL per unit squared displacement (on-policy; fixed-prefix gives 20.9, 7.5, 3.7).
With α fixed, rank 64 moves only a quarter as far as rank 4. Per unit of squared displacement it still costs 3.5–18.5× more KL, so the rank effect is there. The two factors multiply: in seed 0, 0.252×18.5≈1.16, the measured KL ratio. Total KL therefore no longer follows rank: rank 64 ends slightly above rank 4 in one seed and well below it in the other two. In this common setting a lower rank does not keep the policy closer. Rewards are similar (0.81–1.00). This is in Appendix C.2.
2. When aligned initialization reverses the KL ordering, with absolute KL and task performance
KL in nats per sequence at matched displacement. GSM8K: full test set, 512 tokens, greedy, round 30, range over seeds; all rows use the same evaluation script.
At 2e-6 the ratio is below one at every round in every seed, from 0.14 to 0.996. The reversal comes from rank 4: aligned probes raise its KL by two to three orders of magnitude over random probes, more than they raise the KL of rank 64. At the default rate the aligned runs reach 6–13 nats, the KL ratios move toward one, and only the Fisher-sensitivity ratio (0.22–0.78) is still below one in all nine cases. Task accuracy does not suffer. At 2e-6, aligned and random probes score the same. At 2e-5 the aligned runs move the policy one to two orders of magnitude further than the random runs and score higher, with fewer truncated answers.