REPA: Residual-Space Pullback for Scale-Consistent Low-Rank Adaptation

Parameter-efficient fine-tuning (PEFT) is widely used to adapt large pretrained models under constrained memory and computational budgets, with Low-Rank Adaptation (LoRA) serving as a common foundation. Existing LoRA variants primarily modify the parameterization, scaling, initialization, or optimization of low-rank updates, but generally do not account for how these updates are transformed after entering a residual stream followed by Root Mean Square Layer Normalization (RMSNorm), where the projected first-order response depends on residual scale. To address this residual-scale mismatch, this paper proposes REPA (Residual-Space Pullback for Scale-Consistent Low-Rank Adaptation). REPA projects the LoRA candidate update onto the subspace orthogonal to the current residual state, where the local RMSNorm Jacobian reduces to a scalar response, and rescales the projected update by the current residual RMS. This construction cancels the resulting inverse-scale modulation while preserving the standard LoRA rank and trainable parameter count. Theoretical analysis establishes exact local realization of the desired projected response, controlled second-order finite-step deviation, and uniqueness and minimum-norm properties of the resulting pullback. Across seven downstream benchmarks, REPA achieves state-of-the-art macro-average accuracy among the evaluated baselines at $81.19$% and achieves optimal or near-optimal performance on every task. Compared with LoRA, REPA improves macro-average accuracy by $2.19$ percentage points with the same trainable parameter count, while requiring only $1.004$ times the full-training time. Controlled residual rescaling from $0.25$ times to $4$ times produces the predicted inverse-scale response for LoRA while keeping REPA near its reference response, confirming the scale-correction behavior.