acceptodds
Withdrawn from ICLR 2027

OReA: Orthogonal Re-Factorization for Low-Rank Adaptation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Low-rank adaptation is a prominent paradigm for the parameter-efficient fine-tuning of large-scale pre-trained models. Conventional LoRA updates weight as . Based on SVD, recent works parameterize the weight update in a three-factor form: . Existing methods differ in how orthogonality is incorporated, ranging from initialization and regularization to manifold-based optimization. In this paper, we propose OReA, which restores orthogonality through a product-preserving QR–SVD re-factorization at designated training steps. Moreover, we propose a nonlinear variant algorithm, OReA+, to augment the representation with a nonlinear complementary branch. We establish the shared rank- approximation limit of the linear forms with a common frozen reference and bounds for the structured nonlinear extension. Extensive experiments across diverse downstream tasks validate that our method consistently outperforms state-of-the-art methods. Code is available at https://anonymous.4open.science/r/OReA-2BE9.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
LoRA, fine tuning, LLM

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Rry4OpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes OReA, a new orthogonalization-based LoRA method. The authors argue that an SVD-inspired decomposition can address certain limitations of standard LoRA by separating scaling and directional components and better exploiting the geometric structure of low-rank representations. OReA introduces a computationally efficient re-factorization procedure that restores exact orthogonality, while preserving the represented weight update.

In addition, the authors propose OReA+, which incorporates a nonlinear component to increase the expressive capacity of the low-rank adaptation. Although the nonlinear features are constructed to be orthogonal to the original linear factors, they are not necessarily mutually orthogonal. The authors provide a theoretical analysis of the nonlinear component and derive a least-squares correction to account for potential changes in its contribution.

Critical strengths

The proposed methodology is technically interesting, particularly the efficient re-factorization procedure that enforces exact orthogonality while preserving the represented weight update. However, my main concerns are the lack of a clear motivation for the specific approach adopted by OReA compared with existing orthogonality-based methods and, more importantly, the unusually weak LoRA baseline, which substantially undermines the credibility of the reported empirical improvements.

Critical weaknesses

1. Insufficient motivation for the proposed approach relative to existing methods

I feel that there is a gap between the general motivation for orthogonalization, which is supported by prior literature, and the justification for the specific approach proposed in OReA. Although the paper discusses differences from existing methods, it does not sufficiently establish which limitations of these methods motivate OReA's particular re-factorization strategy or why this strategy should yield better results.

For instance, if maintaining exact orthogonality is the key advantage, PiSSA does not guarantee orthogonality throughout training but still achieves performance close to OReA on Qwen2.5-7B-Instruct (84.43 versus 84.88). This observation make it difficult to understand what specific limitation of existing methods OReA is intended to overcome and why its particular approach should provide a meaningful advantage.

2. Experimental improvements and the strength of the LoRA baseline

A primarily empirical justification for the proposed methodology could be sufficient, but I do not find the current experimental evidence convincing. The standard LoRA baseline reported for LLaMA-3.1-8B-Instruct appears unusually weak compared with results reported in several other papers. This discrepancy is not limited to a single study: EVA (2025), StelLA (2025), and Efficient Orthogonal Fine-Tuning with Principal Subspace Adaptation report substantially higher standard LoRA results on overlapping commonsense reasoning benchmarks.

The following table compares the standard LoRA baselines reported in these papers:

BenchmarkOReA paperEVAStelLAPrincipal Subspace Adaptation
ARC-C52.9084.8078.7080.29
ARC-E72.6494.2090.8490.95
BoolQ67.12—75.1673.18
OBQA65.6090.1087.0084.00
PIQA65.8390.3088.1485.31
SIQA67.2083.0080.1874.36

An additional observation concerns the improvements over the standard LoRA baseline. In the original StelLA paper, the reported eight-task average increases from 85.27 for LoRA to 86.72 for StelLA, an improvement of 1.45 percentage points. In contrast, OReA reports an increase from 65.22 for LoRA to 80.98 for OReA on its six-task average, an improvement of 15.76 percentage points. Although these gains are measured under different experimental conditions, I find the magnitude of this discrepancy particularly surprising.

While I recognize that experimental settings differ across studies, discrepancies of this magnitude remain concerning. It is the authors' responsibility to provide sufficiently strong and properly tuned baselines. In my view, the unusually weak LoRA baseline is a major weakness that undermines the credibility of the reported improvements and, consequently, the empirical justification for OReA.

Questions for authors
  1. LoRA baseline performance: Could the authors explain the substantial gap between the standard LoRA baseline reported in this paper and the considerably stronger results reported in several previous studies on overlapping benchmarks?

  2. Justification of the proposed approach: Could the authors clarify which specific limitations of existing orthogonality-based methods, such as StelLA, motivate OReA's re-factorization strategy, and why this approach should be expected to provide better performance?

  3. Nonlinear extension: The results suggest that the nonlinear component introduced in OReA+ provides additional performance improvements over OReA. Have the authors considered incorporating a similar nonlinear component into an existing orthogonality-based baseline, such as StelLA, assuming such an extension is technically feasible?

Figure 2 should be revised, as the plots contain substantial empty space and some curves are barely visible. The y-axis limits should be adjusted to better focus on the relevant value ranges, with appropriately spaced and labeled ticks to improve readability.

Reviewer_RAzGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes OReA, which represents a low-rank weight update as USV⊤USV^\top. It periodically uses QR decomposition and SVD to restore orthonormal factors and a diagonal core while preserving the current update matrix. OReA+ adds a second branch by applying nonlinear functions to the factors and projecting the results onto the orthogonal complements of their column spaces. Both variants can be merged into the pretrained weights for inference.

The analysis establishes preservation properties and approximation-error bounds. Experiments evaluate the methods on synthetic matrix recovery, commonsense reasoning, and language understanding.

Critical strengths
  1. The re-factorization restores orthonormal factors while preserving the current linear update matrix. The procedure is simple and easy to implement.

  2. The complementary branch adds relatively few parameters and remains mergeable into the pretrained weights. It therefore expands the adapter without adding inference operations after merging.

  3. The ablations provide useful evidence for the proposed components. The added branch also improves StelLA in Figure 3, p. 8, and Figure 8, p. 22, examines re-factorization and extended training.

Critical weaknesses
  1. Small gains from OReA+ over OReA. The average improvement in Table 1, p. 7, is modest, and the GLUE results in Table 7, p. 19, are nearly identical. These comparisons leave the practical value of the added branch insufficiently established.

  2. Rationale for the nonlinear mapping. Equation (8), p. 4, explains how the projection reduces overlap between the branches. However, the choice of an elementwise square to generate the additional directions, and its value relative to other simple choices, need a clearer explanation.

  3. Benefit relative to training cost. Table 8, p. 21, reports training times for OReA and LoRA, but does not include OReA+. Given the small gains over OReA, it remains unclear when the extra branch is worth its additional computation.

Questions for authors
  1. It would be better if the authors use the existing repeated runs to show the gains of OReA+ over OReA together with their variability, and explain which settings benefit most from the added branch.

  2. Could you extend the existing timing comparison to include OReA+ on one representative task, so that its gain over OReA can be considered alongside the additional training cost?

  3. What motivates the elementwise square in Eq. (8), and how much does the benefit of the complementary branch depend on this choice? An explanation based on existing evidence or a simple comparison would be helpful.

I appreciate the simplicity of the method, its implementation, and the positive results in the ablations. If the response clarifies the practical benefit of OReA+ relative to its cost and explains the main design choice, I would be open to raising my score.

Reviewer_rF58OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes OReA, a low-rank adaptation method that parameterizes the weight update as ΔW=USV⊤\Delta W=USV^\top, where UU and VV are column-orthogonal and SS is diagonal. Rather than directly optimizing on a manifold, OReA alternates unconstrained gradient updates with a QR–SVD re-factorization that restores orthogonality and diagonality while preserving the current weight-update matrix. The paper further proposes OReA+, which adds a complementary branch constructed from nonlinear transformations of UU and VV. Theoretical analysis characterizes the shared rank-rr approximation limit of OReA and standard LoRA and gives bounds for the rank-at-most-2r2r OReA+ family. Experiments on synthetic matrix recovery, commonsense reasoning, and GLUE report improvements over several LoRA variants.

Critical strengths
  • The product-preserving QR–SVD step is simple and intuitive. It separates ordinary unconstrained optimization from restoration of an SVD-like representation without modifying the current linear update matrix.
  • The paper provides useful analysis, including synthetic recovery, learning-rate/rank sensitivity, effective-rank analysis, re-factorization-frequency ablations, and training-time measurements.
  • Both OReA and OReA+ remain integratable into the pretrained weight at inference, avoiding additional inference modules.
Critical weaknesses
  • Insufficient comparison and positioning relative to closely related LoRA optimization methods
    • Several highly relevant methods are missing from either the empirical comparison or the related-work discussion. RefLoRA [A] is particularly close because it also addresses the non-uniqueness of low-rank factorization through repeated re-factorization during optimization, selecting an equivalent factorization to obtain more balanced and consistent updates. SDS-LoRA [B] directly studies the geometric optimization issue motivating OReA, showing that singular values can induce anisotropic scaling of LoRA gradients and proposing a parameterization that removes this scaling from the backward pass. LoRA-RITE [C] and LoRA-Pro [D] similarly target factorization-dependent optimization and gradient quality. In addition, PoLAR [E], Riemannion [F], and LoRA-S [G] are already cited by the paper but are not experimentally compared. StelLA [H] is included and provides a useful geometric baseline, but it does not cover these other closely related approaches. These methods and the lack of discussions and comparisons against them make it difficult to assess whether OReA provides a distinct optimization advantage over existing reparameterization, re-factorization, and geometric optimization approaches.
  • The theoretical analysis does not establish the claimed optimization advantage
    • The paper correctly shows that OReA and standard LoRA have the same rank-rr representation capacity under a common frozen reference. The product-preserving QR–SVD lemma also only guarantees that the current update matrix is unchanged by re-factorization. Neither result explains why subsequent optimization should converge faster or reach better solutions. Moreover, for ΔW=USV⊤\Delta W=USV^\top, gradients of the directional factors still depend on the scale matrix SS. Thus, restoring orthogonality does not by itself establish that magnitude-direction coupling or anisotropic gradient scaling has been removed. This is particularly important in light of SDS-LoRA [B], which explicitly analyzes this phenomenon. An analysis of conditioning, gradient isotropy/alignment, or convergence would better support OReA's central optimization claim.
  • The reason for OReA+'s improvement (possibly due to increased effective rank)
    • Although OReA+ is described as a nonlinear extension, the nonlinearity is applied to the parameter factors rather than to the model input, so the resulting adapted layer remains linear in the input. The paper itself bounds the resulting update by rank 2r2r, and its effective-rank analysis shows that OReA+ indeed uses more than rr effective directions. Therefore, gains over rank-rr OReA/LoRA may partly arise simply from increased update capacity. Comparisons against rank-2r2r LoRA/OReA or a parameter- and rank-matched two-branch linear baseline are needed to isolate the benefit of the proposed complementary construction.
  • Efficiency and empirical scope are not fully convincing
    • OReA introduces approximately 6–42% additional training time over standard LoRA in the reported GLUE experiments. The timing comparison does not include closely related optimization methods such as StelLA [H], RefLoRA [A], PoLAR [E], or Riemannion [F], and the complete cost of the two-phase OReA+ procedure is not clearly characterized. Moreover, the main downstream evaluation is limited to four GLUE tasks and 7–8B models on commonsense reasoning. Broader generative or non-language experiments would better support the generality of the claims.

[A] Zhang et al., RefLoRA: Refactored Low-Rank Adaptation for Efficient Fine-Tuning of Large Models. NeurIPS 2025.

[B] Oh et al., SDS-LoRA: Overcoming Anisotropic Gradient Scaling in Low-Rank Adaptation. arXiv 2606.16454.

[C] Yen et al., LoRA Done RITE: Robust Invariant Transformation Equilibration for LoRA Optimization. ICLR 2025.

[D] Wang et al., LoRA-Pro: Are Low-Rank Adapters Properly Optimized? ICLR 2025.

[E] Lion et al., PoLAR: Polar-Decomposed Low-Rank Adapter Representation. NeurIPS 2025.

[F] Bogachev et al., LoRA meets Riemannion: Muon Optimizer for Parametrization-independent Low-Rank Adapters. ICLR 2026.

[G] Zheng and Wu, LoRA-S: An Efficient Low Rank Adaptation Scheme via Sylvester Equation. ICLR 2026.

[H] Li et al., StelLA: Subspace Learning in Low-rank Adaptation using Stiefel Manifold. NeurIPS 2025.

Questions for authors
  • Could the authors compare directly with RefLoRA [A] and SDS-LoRA [B]? RefLoRA seems especially close to OReA's use of re-factorization to improve optimization, while SDS-LoRA directly analyzes whether orthogonal subspace representations eliminate singular-value-dependent gradient distortion.
  • Could the authors additionally compare against LoRA-RITE [C], LoRA-Pro [D], PoLAR [E], Riemannion [F], and LoRA-S [G] under matched models, target modules, parameter budgets, and optimization budgets?
  • Since OReA and standard LoRA represent the same rank-rr update class, can the authors provide evidence explaining why QR–SVD re-factorization improves the optimization trajectory—for example through gradient conditioning, singular-value distributions, gradient alignment with full fine-tuning, or convergence analysis?
  • Does periodic QR–SVD re-factorization reduce the anisotropic gradient-scaling phenomenon identified by SDS-LoRA [B]? In particular, how do the conditioning and effective rank of OReA's gradients compare with standard LoRA and SDS-LoRA?
  • Could the authors compare OReA+ against rank-2r2r LoRA/OReA or another capacity-matched linear baseline? This would help determine whether its gains arise from the specific complementary nonlinear construction or simply from access to more effective rank.
  • Could the authors report the full OReA+ training-time/memory overhead and compare efficiency with the closest optimization-focused alternatives rather than only standard LoRA?

For more details, please refer to the weaknesses section above.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.