- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
REPA projects a low-rank update orthogonally to the residual state and multiplies it by the residual RMS. At an interface followed by RMSNorm, this cancels the local inverse-scale response. The experiments use Qwen2.5-1.5B-Instruct with adapters in four terminal MLP down-projections.
Critical strengths
-
The geometric correction has a clear local derivation and a direct mechanism check. Theorems 1–3 distinguish the Jacobian response, finite-step behavior, and unique/minimum-norm solution, including RMSNorm's epsilon. In Appendix B.4, rescaling recorded residuals by 0.25–4 with the candidate update fixed changes LoRA's normalized response but leaves REPA's nearly unchanged, supporting the specific scale-correction claim.
-
The measured training overhead is small. REPA adds no trainable parameters and increases full-training time by approximately 0.4% over LoRA in the reported rank-8, FP32 setup (Table 3). The proposed correction is therefore inexpensive in the tested setting.
Critical weaknesses
-
The full correction has not shown an accuracy advantage over simpler variants. On its three-task subset, Table 7 reports best-validation averages of 87.44 for RMS-only, static-RMS projection, and full REPA, versus 87.11 for both the raw update and projection alone. The authors acknowledge this tie. The projection has a clear geometric role, but its added value for accuracy remains unestablished; these controls are needed on the main evaluation tasks.
-
The main comparison does not separate REPA's benefit from extra tuning. Appendix B uses a common baseline learning rate and, where applicable, scaling coefficient, while REPA chooses them per dataset. On MNLI, REPA uses 1e-4 and alpha 32, compared with 5e-5 and alpha 16 for LoRA and rsLoRA. Selection uses separate development data, but equally tuned baselines are still needed. Table 2's 0.81-point mean advantage over rsLoRA also lacks seed-level uncertainty, making the size and reliability of the gain difficult to assess.
Questions for authors
- Does REPA retain its advantage when LoRA and rsLoRA receive the same per-task tuning budget?
- On the seven main evaluation tasks, how do RMS-only and static-RMS projection compare with full REPA under matched tuning? Please report paired results across seeds, rather than only the three-task best-validation averages in Table 7.