acceptodds
Under review as a conference paper at ICLR 2027

RCA: Fisher-Residual Low-Rank Adaptation for RLVR Reasoning

Abstract

Reinforcement Learning with Verifiable Rewards (RLVR) has become an effective post-training paradigm for improving language-model reasoning. Existing RLVR work has primarily studied reward sources, rollout sampling, KL regularization, etc. In this paper, we study a complementary question: how does RLVR act on the local parameter space of an adapted model, and which adapter update directions provide higher reward improvement per unit policy-change cost? We approach this question through the local Fisher geometry induced by KL-aware RLVR. Our results show that, under a reward–curvature separation condition, updates in the bilateral Fisher-residual channel space can achieve higher reward-per-curvature efficiency than equal-rank selectors that include high-curvature modes. Inspired by this conclusion, we introduce RCA (Residual-Channel Low-Rank Adaptation), a Fisher-guided Low Rank Adapter (LoRA) method for RLVR reasoning. RCA first estimates layerwise Fisher input and output channels with a calibration-time Kronecker-Factored Approximate Curvature (KFAC) approximation. The resulting projectors define a residual-channel low-rank update constraint. Empirically, RCA improves reasoning reliability across pass@1–pass@128 compared with full-finetuning RL and representative LoRA variants, while maintaining a LoRA-sized trainable parameter footprint.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Large Language Models, Parameter-Efficient Fine-Tuning, Reinforcement Learning, Reasoning, Geometric Analysis

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_VEZ9OpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Residual-Channel Adaptation (RCA), a Fisher-guided low-rank adaptation method for RLVR. Motivated by the local KL geometry of policy optimization, the authors argue that avoiding high-Fisher-curvature directions can improve reward gains under a fixed policy-change budget. RCA uses a calibration-time Kronecker-Factored Approximate Curvature (KFAC) approximation to identify the principal input and output Fisher directions of each adapted layer. It then constrains LoRA updates to the bilateral residual subspace through orthogonal projections, while preserving the low-rank parameterization. The paper first derives conditions under which Fisher-residual updates achieve higher reward improvement per unit KL cost. Then, it empirically validates the claimed performance across different language models on several benchmarks.

Critical strengths
  1. Simple and effective method: The bilateral projection is straightforward to implement, preserves the low-rank structure, and requires no modification to the underlying RLVR optimizer.
Critical weaknesses

Critical Weaknesses

  1. Insufficient empirical evidence. Standard LoRA exhibits near-zero performance on several reasoning benchmarks, raising serious concerns about the reliability of the baseline implementation and training configuration. RCA also shows negligible improvements over stronger geometry-aware PEFT methods (e.g., only 0.8 percentage points over GeoRA). Furthermore, the paper omits several relevant PEFT baselines, including MiLoRA [1], DoRA [2], LoRA-RLPO and LoRA-RLMO [3], and MiSS [4], making it difficult to establish the proposed method's advantage over existing approaches.

  2. Gap between theory and the proposed algorithm. Lemma 3.1 and Theorem 3.7 establish KL-efficiency guarantees for linear update subspaces without considering rank constraints. However, RCA restricts updates to a low-rank LoRA parameterization, for which these guarantees do not necessarily hold. The theoretical results therefore do not directly justify the proposed algorithm.

  3. Figure readability. Figures 2 and 3 are difficult to read due to the small font sizes, crowded plots, and close curves.

References

[1] Wang, H., et al. (2025). MiLoRA: Harnessing Minor Singular Components for Parameter-Efficient LLM Finetuning. NAACL 2025.

[2] Liu, S.-Y., et al. (2024). DoRA: Weight-Decomposed Low-Rank Adaptation. ICML 2024.

[3] Zhang, R., Zhu, J., Zhu, H., & Shi, L. (2026). Geometry-Preserving Orthonormal Initialization for Low-Rank Adaptation in RLVR. ICML 2026.

[4] Yin, Q., et al. (2026). Evaluating Parameter Efficient Methods for RLVR. ICML 2026.

Questions for authors

I genuinely hope that these questions can help the authors improve this paper.

1. Baseline configurations and training stability. A fair comparison should not simply use identical hyperparameters for all methods, but should ensure that each baseline is evaluated under properly tuned configurations with comparable tuning budgets. Prior studies have demonstrated that cosine learning-rate decay can substantially stabilize MiLoRA and partially mitigate the instability of PiSSA [1]. Moreover, Mind Lab [2] showed that standard OLoRA can suffer training collapse under RLVR, whereas OLoRA-tail, which uses minor singular directions without singular-value scaling, achieves stable training. Standard LoRA has also been shown to train reliably under RLVR [1, 3, 4], making its near-zero performance in this paper particularly concerning. Could the authors clarify whether the baselines were independently tuned, particularly their learning-rate schedules, and other method-specific hyperparameters? The authors should carefully revisit their baseline configurations to ensure a fair comparison.

2. Sensitivity to response length. The maximum training response length is limited to only 3,072 tokens, substantially shorter than the 16K-token budgets used in recent RLVR studies [1, 3]. While RCA may perform better under this restricted budget, it remains unclear whether its advantage persists with longer reasoning trajectories. Notably, the failed baselines in Table 2 reach the maximum response length, suggesting that truncation may contribute to their poor performance. Could the authors clarify the maximum response lengths used during both training and evaluation, and report the corresponding truncation rates? More importantly, could they evaluate RCA and its baselines under longer response budgets (e.g., 8K or 16K tokens) to verify whether the reported improvements persist?

3. Theoretical guarantees under low-rank constraints. Mentioned in weakness 2, could the authors clarify whether the theoretical KL-efficiency advantage continues to hold under this rank constraint, or provide additional theoretical?

4. Consistency of LoRA baselines. What is the difference between the standard LoRA baseline in Figure 2 and the "Shared-config LoRA" baseline in Tables 11 and 14? Standard LoRA achieves near-zero performance on several benchmarks in Figure 2, whereas Shared-config LoRA achieves 24.2% pass@1 in the appendix. Please explain this discrepancy and share detailed training logs. Again, it is really counter-intuitive that LoRA will collapse in training. Please check and share the training implementations.

5. Missing citations and insufficient discussion of principal subspaces. The paper overlooks several relevant PEFT studies, including MiLoRA [1], OLoRA-tail [2], LoRA-RLPO/RLMO [3], and MiSS [4]. More importantly, different studies adopt substantially different definitions of principal and off-principal directions [1,2,3,5], yet the paper does not adequately discuss these distinctions.

References

[1] Zhang, R., Zhu, J., Zhu, H., & Shi, L. (2026). Geometry-Preserving Orthonormal Initialization for Low-Rank Adaptation in RLVR. ICML 2026.

[2] Mind Lab. (2026). On the Scaling of PEFT: Towards Million Personal Models of Trillion Parameters.

[3] Yin, Q., et al. (2026). Evaluating Parameter Efficient Methods for RLVR. ICML 2026.

[4] Schulman, J., & Thinking Machines Lab. (2025). LoRA Without Regret.

[5] Zhu et al. (2025). The Path Not Taken: RLVR Provably Learns Off the Principals.

Reviewer_eyyVOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes RCA, a Fisher-guided residual-channel LoRA method for reinforcement learning with verifiable rewards. RCA uses a calibration stage with KFAC to identify high-curvature input and output channels, then constrains LoRA updates to the bilateral residual-channel space. The paper provides a local KL-efficiency analysis under a reward–curvature separation condition and evaluates RCA on mathematical reasoning and knowledge-retention benchmarks. The reported results suggest that RCA can improve pass@k performance while retaining a LoRA-sized trainable parameter footprint.

Critical strengths
  1. The paper presents a clear Fisher-guided residual-channel LoRA method for RLVR.

  2. It provides useful geometric analyses and empirical controls beyond the main results.

Critical weaknesses
  1. The evaluated models are too old, and the coverage of reasoning models is too limited. Among the evaluated backbones, only DeepSeek-R1-Distill-Qwen-1.5B is a reasoning model. The method should be extended to Qwen3 or Qwen3.5 reasoning models, at least a smaller model, to verify whether the proposed approach is effective on modern reasoning models.

  2. The paper provides insufficient intuitive explanation of what the high-curvature directions are updating and why they induce large KL perturbations, corresponding to large changes in the output probabilities, while having limited effect on accuracy. It is unclear whether this phenomenon normally occurs during RL training or whether these directions correspond to events that are already relatively low-probability. The paper also does not explain what the residual-channel space is updating, why restricting updates to this space resolves the problem, or why this mechanism leads to such large performance gains. A clearer intuitive explanation is needed to connect the channel geometry, output-probability changes, and accuracy improvements.

  3. The method is validated only in the LoRA setting. The paper does not investigate whether the same Fisher-residual restriction is applicable to non-LoRA training settings, such as full fine-tuning or other parameterizations. It is therefore unclear whether the proposed idea is a general update-direction principle or a technique specific to LoRA.

Questions for authors

See Critical Weaknesses.

Reviewer_nAUKOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
2(+1)
Read review
Paper summary

The paper studies parameter-efficient RLVR through the local Fisher/KL geometry. It defines a KL-efficiency functional ΓF(P)=g⊤(PFP)†g\Gamma_F(P)=g^\top(PFP)^\dagger g for linearly restricted updates (Lemma 3.1). Under a Kronecker-factored Fisher proxy F0=Cx⊗CδF_0=C_x\otimes C_\delta, it argues (Thms 3.4, 3.7) that the bilateral residual space, obtained by removing the top-rkr_k input and output eigen-channels, has higher reward-per-curvature efficiency than equal-rank spectral selectors that include blocked modes. This requires a reward–curvature separation condition and a relative approximation bound ε<1\varepsilon<1. The resulting method, RCA, estimates the leading eigenspaces of the KFAC factors in a one-time calibration pass and trains LoRA factors constrained as ΔW=QδBAQx\Delta W=Q_\delta BAQ_x. Experiments use VERL/GRPO on DeepScaleR with four backbones and report:

  • pass@1–128 on AIME24/25 and AMC23;
  • knowledge retention on MMLU, ARC-C, HellaSwag and GPQA;
  • an rkr_k ablation, basis and side controls;
  • appendix geometry audits.

The paper reports that RCA outperforms FFT and several LoRA variants, and that standard LoRA collapses.

Critical strengths

None. The question posed (which adapter directions are efficient under the local KL geometry of RLVR) is reasonable, but neither the theory nor the experiments provide reliable evidence about it (see Weaknesses).

Critical weaknesses

W1. The theory's premises do not hold, and the theory matches neither the experiments nor the code. This matters because the contribution is framed as theory-guided.

  • The central claim, that Fisher alignment is the key, is contradicted by OPLoRA as they reported. RCA and OPLoRA use the same bilateral projection and differ only in the basis (KFAC vs. weight SVD, l.747–750). On the paper's own diagnostic (product overlap, l.795–797), OPLoRA's basis is indistinguishable from a random one (Table 3). The paper attributes the random basis's collapse to this misalignment (l.461–463), yet OPLoRA trains stably and stays close to RCA (Fig. 2, Table 12), even with a learning rate shared by all methods and no reported tuning. The repository README states that RCA originally used the weight-SVD basis; the switch to KFAC is never justified.
  • Thm 3.7 avoids the relevant comparisons. It only compares against equal-rank Fisher-basis selectors that contain blocked modes, which excludes FFT, LoRA and OPLoRA. Since the KL-limited gain of Lemma 3.1 can only grow with the feasible set, and RCA ⊂ LoRA ⊂ FFT, the theory itself predicts FFT ≥ LoRA ≥ RCA, the opposite of the reported results. ΓF\Gamma_F is never measured for any baseline, so the experiments establish no link between this efficiency and performance.
  • The required bound ε<1\varepsilon<1 is never measured, as the authors acknowledge (l.1216–1219).
  • Different KLs are conflated. The reference-KL constraint (Eq. 1), the per-step PPO/GRPO clipping (l.108–110) and the calibration curvature are treated as one quantity. In the released code, the default calibration is the NLL on 32 prompts, not the policy Fisher of Eq. 2.

W2. The experiments are not reliable. This matters because the headline gains are measured against implausible baselines and supported by tautological evidence.

  • The LoRA collapse (l.376–378) is unexplained and contradicts the source cited for it. Yin et al. (2025) do not report LoRA collapse, and Schulman & Thinking Machines (2025) report that LoRA matches FFT in RL. LoRA-based RLVR is widely reported to train successfully.
  • The leakage metric is zero by construction (Cor. D.2) but is used as evidence of superiority. In the released code, η\eta is computed with the same bases used for the projection, so the random control would also give ≈0\approx0, and its value is omitted for no reason (Tables 2, 14). But this metrics is very important for their theory and methodology.
  • The released code does not match the paper. The Rayleigh–Ritz step of App. C.1 is not implemented, the default learning rate differs from Table 5, and the implementations of the PEFT baselines have been removed.
  • Many reported numbers have unclear provenance. For example, the Table 10 caption reads "The ± summaries are retained as reported; no additional seed count or uncertainty definition is inferred" (see also l.1187, l.1300, l.1322–1323).
Questions for authors
  1. Please report RCA with the weight-SVD basis, with the random basis, and OPLoRA, under the same tuned settings and with multiple seeds. Why does OPLoRA train stably while the random basis collapses?
  2. For the collapsed LoRA runs, please provide training curves and a learning-rate sweep. Which result in Yin et al. (2025) supports LoRA collapse in RLVR?
  3. How is η\eta computed for each method, and what is its value for the random-basis run?
  4. Can you measure or estimate ε\varepsilon on the actual layers?
  5. How many seeds underlie the ± values in Tables 10, 11 and 14, and what does "retained as reported" refer to?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.