acceptodds
Under review as a conference paper at ICLR 2027

REPA: Residual-Space Pullback for Scale-Consistent Low-Rank Adaptation

Abstract

Parameter-efficient fine-tuning (PEFT) is widely used to adapt large pretrained models under constrained memory and computational budgets, with Low-Rank Adaptation (LoRA) serving as a common foundation. Existing LoRA variants primarily modify the parameterization, scaling, initialization, or optimization of low-rank updates, but generally do not account for how these updates are transformed after entering a residual stream followed by Root Mean Square Layer Normalization (RMSNorm), where the projected first-order response depends on residual scale. To address this residual-scale mismatch, this paper proposes REPA (Residual-Space Pullback for Scale-Consistent Low-Rank Adaptation). REPA projects the LoRA candidate update onto the subspace orthogonal to the current residual state, where the local RMSNorm Jacobian reduces to a scalar response, and rescales the projected update by the current residual RMS. This construction cancels the resulting inverse-scale modulation while preserving the standard LoRA rank and trainable parameter count. Theoretical analysis establishes exact local realization of the desired projected response, controlled second-order finite-step deviation, and uniqueness and minimum-norm properties of the resulting pullback. Across seven downstream benchmarks, REPA achieves state-of-the-art macro-average accuracy among the evaluated baselines at % and achieves optimal or near-optimal performance on every task. Compared with LoRA, REPA improves macro-average accuracy by percentage points with the same trainable parameter count, while requiring only times the full-training time. Controlled residual rescaling from times to times produces the predicted inverse-scale response for LoRA while keeping REPA near its reference response, confirming the scale-correction behavior.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
parameter-efficient fine-tuning, low-rank adaptation (LoRA), large language models, foundation model adaptation, RMSNorm, normalization, residual connections

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_2WFpOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

REPA projects a low-rank update orthogonally to the residual state and multiplies it by the residual RMS. At an interface followed by RMSNorm, this cancels the local inverse-scale response. The experiments use Qwen2.5-1.5B-Instruct with adapters in four terminal MLP down-projections.

Critical strengths
  1. The geometric correction has a clear local derivation and a direct mechanism check. Theorems 1–3 distinguish the Jacobian response, finite-step behavior, and unique/minimum-norm solution, including RMSNorm's epsilon. In Appendix B.4, rescaling recorded residuals by 0.25–4 with the candidate update fixed changes LoRA's normalized response but leaves REPA's nearly unchanged, supporting the specific scale-correction claim.

  2. The measured training overhead is small. REPA adds no trainable parameters and increases full-training time by approximately 0.4% over LoRA in the reported rank-8, FP32 setup (Table 3). The proposed correction is therefore inexpensive in the tested setting.

Critical weaknesses
  1. The full correction has not shown an accuracy advantage over simpler variants. On its three-task subset, Table 7 reports best-validation averages of 87.44 for RMS-only, static-RMS projection, and full REPA, versus 87.11 for both the raw update and projection alone. The authors acknowledge this tie. The projection has a clear geometric role, but its added value for accuracy remains unestablished; these controls are needed on the main evaluation tasks.

  2. The main comparison does not separate REPA's benefit from extra tuning. Appendix B uses a common baseline learning rate and, where applicable, scaling coefficient, while REPA chooses them per dataset. On MNLI, REPA uses 1e-4 and alpha 32, compared with 5e-5 and alpha 16 for LoRA and rsLoRA. Selection uses separate development data, but equally tuned baselines are still needed. Table 2's 0.81-point mean advantage over rsLoRA also lacks seed-level uncertainty, making the size and reliability of the gain difficult to assess.

Questions for authors
  1. Does REPA retain its advantage when LoRA and rsLoRA receive the same per-task tuning budget?
  2. On the seven main evaluation tasks, how do RMS-only and static-RMS projection compare with full REPA under matched tuning? Please report paired results across seeds, rather than only the three-task best-validation averages in Table 7.
Reviewer_45hHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the residual-scale mismatch of LoRA updates in Pre-RMSNorm Transformers. The proposed REPA method first projects the candidate LoRA update onto the subspace orthogonal to the current residual state. It then scales the projected update by the current residual RMS before injecting it into the residual stream, thereby compensating for the scale-dependent modulation introduced by RMSNorm.

Critical strengths
  1. The proposed method is theoretically motivated. The paper explicitly distinguishes the responses of RMSNorm along the residual-parallel and residual-orthogonal directions and provides theoretical justification for the resulting projection and scaling operations.
  2. The empirical results are generally positive. REPA consistently outperforms standard LoRA across all seven evaluated tasks. These results are consistent with the hypothesis that standard LoRA is affected by variations in residual scale and suggest that REPA can alleviate this issue to some extent.
Critical weaknesses
  1. REPA uses dataset-specific learning rates and scaling coefficients α\alpha, whereas the baselines use a shared learning rate and α\alpha across datasets. This discrepancy may lead to an unfair comparison, as part of REPA’s improvement could be attributed to additional hyperparameter tuning rather than the proposed method itself.
  2. The experiments are limited to Qwen2.5-1.5B-Instruct, use only 500 training examples per task, and apply the adapters only to the final four Transformer layers. It therefore remains unclear whether the method generalizes to larger models, more comprehensive adaptation settings, or broader datasets.
  3. The ablation study suggests that most of the performance improvement comes from simple RMS-based scaling rather than the residual-orthogonal projection. Therefore, the current experiments mainly show that appropriately scaling the update can be beneficial, but they do not sufficiently establish that the complete residual-space pullback mechanism is necessary for achieving the reported gains.
Questions for authors
  1. Could the authors provide results on larger models and larger datasets?
  2. Could the authors report the additional computational and memory overhead introduced by REPA during both training and inference?
Reviewer_sjzTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes REPA, a modification of low-rank adaptation designed to account for the transformation of adapter updates by RMSNorm in a residual stream. REPA projects the adapter output onto the subspace orthogonal to the current residual state and scales it by the residual RMS. The paper derives local and finite-step properties of this construction and evaluates it on seven classification tasks using Qwen2.5-1.5B-Instruct with 500 training examples per task. The reported results show higher average accuracy than the evaluated baselines, with small measured training-time overhead in the studied configuration.

Critical strengths
  1. The method has a clear geometric motivation: it explicitly accounts for the local response of RMSNorm rather than treating the adapter output as an unchanged perturbation. The analytical characterization makes the intended scale compensation precise.

  2. The paper includes informative component ablations and measurements of residual-parallel update energy. These analyses help distinguish the proposed geometric mechanism from simpler scaling effects, even though they do not yet establish the benefit of the full construction.

Critical weaknesses

W1. Unequal hyperparameter tuning confounds the reported performance gains

The main comparison does not provide an equal hyperparameter-tuning budget across methods. Appendix B.1 states that baselines use a common learning rate of (5×10−5)(5\times10^{-5}) and, where applicable, (α=16)(\alpha=16), whereas REPA uses dataset-specific learning rates and scaling coefficients selected on development data. This asymmetry is particularly consequential because REPA itself modifies the effective scale of adapter outputs. Without equally tuned LoRA and rsLoRA baselines, the experiments cannot isolate the benefit of residual-space correction from that of better scaling and optimization settings. Although REPA improves the seven-task average over LoRA by 2.19 percentage points, its advantage over rsLoRA is only 0.81 points, making a tuning-matched comparison essential to substantiate the claimed improvement.

W2. The ablations do not establish the benefit of the full proposed mechanism

The component analysis does not demonstrate that the combination of residual-orthogonal projection and dynamic RMS scaling is necessary for the observed accuracy gains. In Table 7, RMS-only scaling, static layer-wise RMS scaling with projection, and full REPA all achieve the same three-task average validation accuracy of 87.44%, while projection alone matches the uncorrected baseline at 87.11%. Appendix B.4 further reports that only approximately 0.11–0.15% of LoRA update energy lies in the residual-parallel direction, consistent with the negligible empirical effect of projection. The controlled-rescaling experiment verifies the predicted normalization response, but does not establish that enforcing this response improves downstream learning. Consequently, the current evidence does not show an advantage over a simpler static scaling strategy, despite REPA introducing state-dependent operations that generally prevent static weight merging. Broader, tuning-matched ablations on held-out final-evaluation data are needed to establish the practical value of the complete construction.

Questions for authors
  1. Clarification affecting the main comparison: Were LoRA and rsLoRA given the same dataset-specific learning-rate and scaling searches as REPA? If so, please provide the search spaces, selection protocol, and corresponding results; otherwise, can you provide a tuning-matched comparison?

  2. Evidence affecting the mechanism claim: Does full REPA outperform RMS-only scaling and static layer-wise RMS scaling with projection on held-out final-evaluation data under matched hyperparameter selection? In light of Table 7 and the small residual-parallel energy reported in Appendix B.4, what empirical evidence attributes an improvement specifically to dynamic compensation or projection?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.