- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper observes that under RLVR the LoRA input-side factor barely moves, so its initialization determines which directions LoRA can learn. It proposes REAL, which selects the input subspace from per-group reward-contrast gradients via a trace-maximization criterion eigenvectors of and freezes during training, with two-sided random compression for efficiency. Experiments on Qwen3 and DeepSeek-R1-Distill models show consistent accuracy gains over seven PEFT baselines and large reductions in completion length.
Critical strengths
-
A well-supported diagnosis of why LoRA underperforms under RLVR. The paper identifies a specific mechanism (near-invariance of the input subspace, second-order rotation) and verifies it with both measurement (Figure 2) and theory (Theorem 1). This reframes the problem: the bottleneck is not LoRA's capacity but the choice of input subspace at initialization, which makes REAL a principled remedy rather than an ad hoc fix.
-
A principled and efficient method with strong empirical validation. REAL derives its criterion from a clear objective, proves the optimum (Proposition 1), and contrasts its per-group accumulation with LoRA-GA's aggregation (Eq. 8). Two-sided random compression makes it practical at scale. The reward-gradient coverage analysis (61.0% vs. 18.5% for the strongest baseline) directly validates the mechanism on held-out problems. Gains are consistent across three model sizes and two distilled models with roughly half of LoRA's trainable parameters.
Critical weaknesses
-
The authors' reasoning and experiments are impressive. However, one point is worth considering: in practice, people still tend not to use LoRA for RL training, and a key reason is that LoRA is more prone to training collapse than full fine-tuning (this occurs both at the beginning of training and when the reward plateaus). Do the authors' theory and method offer any positive insight into explaining or mitigating this collapse phenomenon?
-
The experimental models are all relatively small. I understand that experiments on larger models may not be feasible, but I would appreciate it if the authors could supplement or verify the advantages of this method at higher dimensions.
-
Is this method equally effective for other tasks, such as dense rewards? As described in [1], for denser rewards like OPD, the trajectories in the reward space may differ from those in RL.
-
Could the authors visualize the update trajectories of LoRA's and after PCA projection and discuss them appropriately? As described in [2], they observed that the top-1 update trajectories of some parameters exhibit a certain linear trend.
These are my concern on this paper. If the authors can respond to these questions reasonably and appropriately, I will raise my score to 4.
[1] Learning to Foresee: Unveiling the Unlocking Efficiency of On-Policy Distillation
[2] On Predictability of Reinforcement Learning Dynamics for Large Language Models
Questions for authors
See Critical Weaknesses