acceptodds
Under review as a conference paper at ICLR 2027

Reward-Gradient Aligned Low-Rank Adaptation for Post-Training

Abstract

Reinforcement Learning with Verifiable Rewards (RLVR) has become a prominent post-training paradigm for enhancing the reasoning capabilities of large language models. Low-Rank Adaptation (LoRA) offers a natural way to reduce its training cost. Intuitively, training both low-rank factors should allow LoRA to adapt along the directions that matter for the reward. However, we observe that the input-side factor of LoRA barely changes during RLVR. As a result, the input subspace remains close to its random initialization throughout training. We prove that under standard initialization this subspace rotates only at second order in the cumulative gradient, one order slower than the output-side factor. The random initialization therefore persistently constrains subsequent optimization. Motivated by this finding, we propose REward-Gradient Aligned Low-Rank Adaptation (REAL), which aligns the input subspace with task-specific reward gradients at initialization. REAL contrasts the gradients of correct and incorrect rollouts for each prompt and selects the subspace that retains the most energy of these contrasts across prompts. Since exact selection requires a full weight gradient for every prompt, REAL estimates the subspace by two-sided random compression with memory linear in the layer width. Extensive experiments on the Qwen3 series show that REAL outperforms seven PEFT baselines and exceeds LoRA by up to 5.90% in average accuracy. With a length-aware selection reward, REAL also shortens the responses of distilled reasoning models by up to 44.9% while improving their accuracy.

Primary area:
foundation or frontier models, including LLMs
Keywords:
PEFT, RLVR, Reasoning LLMs

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_qgihOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper observes that under RLVR the LoRA input-side factor AA barely moves, so its initialization determines which directions LoRA can learn. It proposes REAL, which selects the input subspace from per-group reward-contrast gradients via a trace-maximization criterion top−rtop-r eigenvectors of �0�)�0�) and freezes AA during training, with two-sided random compression for efficiency. Experiments on Qwen3 and DeepSeek-R1-Distill models show consistent accuracy gains over seven PEFT baselines and large reductions in completion length.

Critical strengths
  1. A well-supported diagnosis of why LoRA underperforms under RLVR. The paper identifies a specific mechanism (near-invariance of the input subspace, second-order rotation) and verifies it with both measurement (Figure 2) and theory (Theorem 1). This reframes the problem: the bottleneck is not LoRA's capacity but the choice of input subspace at initialization, which makes REAL a principled remedy rather than an ad hoc fix.

  2. A principled and efficient method with strong empirical validation. REAL derives its criterion from a clear objective, proves the optimum (Proposition 1), and contrasts its per-group accumulation with LoRA-GA's aggregation (Eq. 8). Two-sided random compression makes it practical at scale. The reward-gradient coverage analysis (61.0% vs. 18.5% for the strongest baseline) directly validates the mechanism on held-out problems. Gains are consistent across three model sizes and two distilled models with roughly half of LoRA's trainable parameters.

Critical weaknesses
  1. The authors' reasoning and experiments are impressive. However, one point is worth considering: in practice, people still tend not to use LoRA for RL training, and a key reason is that LoRA is more prone to training collapse than full fine-tuning (this occurs both at the beginning of training and when the reward plateaus). Do the authors' theory and method offer any positive insight into explaining or mitigating this collapse phenomenon?

  2. The experimental models are all relatively small. I understand that experiments on larger models may not be feasible, but I would appreciate it if the authors could supplement or verify the advantages of this method at higher dimensions.

  3. Is this method equally effective for other tasks, such as dense rewards? As described in [1], for denser rewards like OPD, the trajectories in the reward space may differ from those in RL.

  4. Could the authors visualize the update trajectories of LoRA's AA and BB after PCA projection and discuss them appropriately? As described in [2], they observed that the top-1 update trajectories of some parameters exhibit a certain linear trend.

These are my concern on this paper. If the authors can respond to these questions reasonably and appropriately, I will raise my score to 4.

[1] Learning to Foresee: Unveiling the Unlocking Efficiency of On-Policy Distillation

[2] On Predictability of Reinforcement Learning Dynamics for Large Language Models

Questions for authors

See Critical Weaknesses

Reviewer_S9jMOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Reward-Gradient Aligned Low-Rank Adaptation (REAL) for RLVR post-training. Motivated by the observation that LoRA's input-side factor changes little during training, REAL selects a low-dimensional input subspace using the second moment of per-prompt reward-contrast gradients. They initialize the input lora adapter as the selected subspace and freeze during training. They only train the output adapter. Experiments on Qwen3 and DeepSeek-R1-Distill models report improvements in reasoning accuracy and reductions in response length.

Critical strengths
  1. The method is conceptually simple and has a clear mathematical formulation. The motivation is clear and reasonable.

  2. The paper evaluates REAL across multiple model sizes and reasoning benchmarks, with thorough comparisons against several PEFT methods and ablations.

Critical weaknesses
  1. Proposition 1 maximizes Ep[∥GpP∥F2]\mathbb{E}_p[\|G_pP\|_F^2], whereas the first-order improvement of the shared RLVR objective under full-batch gradient ascent is proportional to ∥Ep[Gp]P∥F2\|\mathbb{E}_p[G_p]P\|_F^2. These quantities differ by the projected gradient variance, Ep[∥(Gp−Ep[Gp])P∥F2]\mathbb{E}_p[\|(G_p-\mathbb{E}_p[G_p])P\|_F^2]. The paper does not establish why maximizing this additional variance term should lead to greater expected reward improvement. Moreover, Table 6 shows only a 0.61 percentage-point improvement over the aggregated-gradient baseline, without reporting standard deviations or statistical significance. Additional theory and more experiments are needed to convince me the paper's design choices.

  2. The paper uses the same learning rate for all PEFT methods, although Appendix G.4 shows that standard LoRA benefits from a higher learning rate. A fair comparison should evaluate each method under properly tuned configurations. Moreover, closely related gradient-based initialization methods, such as LoRA-GA and LoRA-One, are missing from the main comparisons.

  3. The reported length reductions rely on an explicit length-aware reward during initialization, but the paper lacks comparisons with correctness-only REAL or other PEFT methods using similar length rewards. Thus, it remains unclear whether the improvements arise from REAL's subspace selection or simply from the additional length preference.

Questions for authors
  1. Algorithm 1 requires at least two correct and two incorrect responses per group, whereas Appendix E states that all-correct groups can provide length-contrast signals. Were the group-selection criteria modified for the length-aware experiments?

  2. Could the authors report the total training time and computational cost of REAL, including reward-gradient-based initialization, compared with standard LoRA?

  3. Figure 4 evaluates the learned update subspace using reward gradients computed at the initial policy (t=0t=0), rather than the current policy during training. Why is alignment with the initial gradients a meaningful measure of the effectiveness of subsequent RLVR optimization? Besides, the reported high coverage is expected by the construction of initialization. I just don't understand what this figure want to convey.

Reviewer_ecKpOpenReview ↗
Rating
3
Confidence
3(+1)
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies a surprising behavior of LoRA during reinforcement learning with verifiable rewards. In the studied setting, the input subspace available to the LoRA update stays close to its random initialization even when both factors are trained. The paper proposes REAL, which chooses these directions using rewards from correct and incorrect responses before training begins. It then keeps those directions fixed and trains only the other factor. Experiments on three Qwen3 models report better reasoning accuracy, and an additional experiment reports shorter responses.

Critical strengths
  1. I like that the paper starts from an actual observation. If LoRA barely changes its input directions during training, it makes sense to ask whether choosing better directions at the start could help.

  2. The solution is simple and the results are encouraging across three model sizes. It also avoids changing the reinforcement-learning reward during the main training stage.

Critical weaknesses
  1. I am not fully convinced by the reason for selecting directions with large per-prompt gradient energy. Different prompts can favor opposite changes to the same model, so retaining the strength of these conflicting gradients does not necessarily help the overall training objective.

  2. The paper shows that reward-based initialization helps, but it is less clear how much of the gain comes from REAL's specific way of choosing directions. The shorter-response experiment also does not separate the effect of the length-aware initialization from the effect of doing more reinforcement-learning training.

Questions for authors
  1. Figure 2: Interesting observation. But does the input subspace still barely change if A and B are trained with different learning rates, as in LoRA+? If ordinary training choices can make A move more, how serious is the random-initialization problem in practice?

  2. Section 3.2: I understand the motivation for avoiding gradient cancellation across prompts. But isn't some cancellation actually desirable when different prompts produce conflicting gradients? Why should preserving these directions help the overall RL objective? I would be interested in seeing whether the selected subspace improves subsequent optimization, rather than just retaining more initial gradient energy.

  3. Appendix G.3, Table 6: The aggregate-gradient initialization already reaches 46.34, compared with 46.95 for REAL Train and 47.10 for REAL Frozen. Could you clarify whether REAL Gradient uses the full LoRA-GA initialization or only its input-subspace selection? In particular, how are A and B initialized and scaled in REAL Gradient compared with REAL Train?

  4. Table 2: The shorter responses are interesting. But could they result from the additional reinforcement-learning training rather than the length-aware initialization? What happens if you keep the training identical but choose the initial directions using correctness rewards only?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.