- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper proposes OReA, a new orthogonalization-based LoRA method. The authors argue that an SVD-inspired decomposition can address certain limitations of standard LoRA by separating scaling and directional components and better exploiting the geometric structure of low-rank representations. OReA introduces a computationally efficient re-factorization procedure that restores exact orthogonality, while preserving the represented weight update.
In addition, the authors propose OReA+, which incorporates a nonlinear component to increase the expressive capacity of the low-rank adaptation. Although the nonlinear features are constructed to be orthogonal to the original linear factors, they are not necessarily mutually orthogonal. The authors provide a theoretical analysis of the nonlinear component and derive a least-squares correction to account for potential changes in its contribution.
Critical strengths
The proposed methodology is technically interesting, particularly the efficient re-factorization procedure that enforces exact orthogonality while preserving the represented weight update. However, my main concerns are the lack of a clear motivation for the specific approach adopted by OReA compared with existing orthogonality-based methods and, more importantly, the unusually weak LoRA baseline, which substantially undermines the credibility of the reported empirical improvements.
Critical weaknesses
1. Insufficient motivation for the proposed approach relative to existing methods
I feel that there is a gap between the general motivation for orthogonalization, which is supported by prior literature, and the justification for the specific approach proposed in OReA. Although the paper discusses differences from existing methods, it does not sufficiently establish which limitations of these methods motivate OReA's particular re-factorization strategy or why this strategy should yield better results.
For instance, if maintaining exact orthogonality is the key advantage, PiSSA does not guarantee orthogonality throughout training but still achieves performance close to OReA on Qwen2.5-7B-Instruct (84.43 versus 84.88). This observation make it difficult to understand what specific limitation of existing methods OReA is intended to overcome and why its particular approach should provide a meaningful advantage.
2. Experimental improvements and the strength of the LoRA baseline
A primarily empirical justification for the proposed methodology could be sufficient, but I do not find the current experimental evidence convincing. The standard LoRA baseline reported for LLaMA-3.1-8B-Instruct appears unusually weak compared with results reported in several other papers. This discrepancy is not limited to a single study: EVA (2025), StelLA (2025), and Efficient Orthogonal Fine-Tuning with Principal Subspace Adaptation report substantially higher standard LoRA results on overlapping commonsense reasoning benchmarks.
The following table compares the standard LoRA baselines reported in these papers:
| Benchmark | OReA paper | EVA | StelLA | Principal Subspace Adaptation |
|---|---|---|---|---|
| ARC-C | 52.90 | 84.80 | 78.70 | 80.29 |
| ARC-E | 72.64 | 94.20 | 90.84 | 90.95 |
| BoolQ | 67.12 | — | 75.16 | 73.18 |
| OBQA | 65.60 | 90.10 | 87.00 | 84.00 |
| PIQA | 65.83 | 90.30 | 88.14 | 85.31 |
| SIQA | 67.20 | 83.00 | 80.18 | 74.36 |
An additional observation concerns the improvements over the standard LoRA baseline. In the original StelLA paper, the reported eight-task average increases from 85.27 for LoRA to 86.72 for StelLA, an improvement of 1.45 percentage points. In contrast, OReA reports an increase from 65.22 for LoRA to 80.98 for OReA on its six-task average, an improvement of 15.76 percentage points. Although these gains are measured under different experimental conditions, I find the magnitude of this discrepancy particularly surprising.
While I recognize that experimental settings differ across studies, discrepancies of this magnitude remain concerning. It is the authors' responsibility to provide sufficiently strong and properly tuned baselines. In my view, the unusually weak LoRA baseline is a major weakness that undermines the credibility of the reported improvements and, consequently, the empirical justification for OReA.
Questions for authors
-
LoRA baseline performance: Could the authors explain the substantial gap between the standard LoRA baseline reported in this paper and the considerably stronger results reported in several previous studies on overlapping benchmarks?
-
Justification of the proposed approach: Could the authors clarify which specific limitations of existing orthogonality-based methods, such as StelLA, motivate OReA's re-factorization strategy, and why this approach should be expected to provide better performance?
-
Nonlinear extension: The results suggest that the nonlinear component introduced in OReA+ provides additional performance improvements over OReA. Have the authors considered incorporating a similar nonlinear component into an existing orthogonality-based baseline, such as StelLA, assuming such an extension is technically feasible?
Figure 2 should be revised, as the plots contain substantial empty space and some curves are barely visible. The y-axis limits should be adjusted to better focus on the relevant value ranges, with appropriately spaced and labeled ticks to improve readability.