acceptodds
Under review as a conference paper at ICLR 2027

Amortizing Exact Orthogonality: Drift-Triggered Restoration for Orthogonal Fine-Tuning

Abstract

Orthogonal finetuning (OFT) preserves the geometry of pretrained weights, but existing methods construct an exact or approximate orthogonal transform at every optimization step. We show that the construction parameter used for this purpose does not reliably control the orthogonality realized during training: under fixed truncation settings, checkpoints from the reference OFTv2 implementation exhibit defects spanning five orders of magnitude. We introduce Restarted Tangent Orthogonal Finetuning (RTOFT), which replaces the per-step orthogonal construction with the tangent update , where , and restores exact orthogonality only when the measured drift reaches a tolerance . For this update, exactly, so monitoring and restoring before the next forward pass guarantees independently of the loss, data, model, or optimizer; exact orthogonality is therefore guaranteed at the returned model, while every forward pass remains within the prescribed defect budget. Across block-diagonal, subspace, and butterfly parameterizations, RTOFT reduces training time from 1.10–1.58 LoRA's wall-clock to 0.97–1.12. The block variant runs at 0.97 LoRA's wall-clock while exceeding LoRA by points on the six-task GLUE average over five shared seeds. The same drift tolerance transfers across DeBERTaV3, Qwen, and diffusion models, whereas a fixed restoration period realizes substantially different geometry. Matched controls further show that restoration along the optimization path matters, while restoring at every step provides no measurable benefit over less frequent restoration. RTOFT therefore makes realized geometric drift, rather than elapsed optimization steps, the criterion for restoring exact orthogonality.

Primary area:
optimization
Keywords:
parameter-efficient fine-tuning, orthogonal fine-tuning, constrained optimization, manifold optimization, orthogonality, efficient training

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_tL7COpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

OFT preserves the geometric properties of pretrained models, but existing methods require exact or approximate orthogonal transformations at every optimization step, introducing additional computational overhead. The authors observe that the approximation scheme used in OFTv2 does not reliably control the realized orthogonality. To address this issue, they propose RTOFT, which replaces per-step orthogonal construction with inexpensive tangent updates. RTOFT restores exact orthogonality only when the measured drift reaches the predefined threshold τ\tau, thereby amortizing the cost of maintaining orthogonality across optimization steps. Extensive experiments across multiple datasets and models demonstrate that RTOFT achieves competitive or improved downstream performance while substantially reducing the computational overhead of OFT.

Critical strengths
  1. The paper is well motivated and addresses a genuine challenge in OFT. The additional computational overhead required to maintain orthogonality is a key limitation of existing OFT methods, making this an important and practically relevant problem.

  2. RTOFT offers a novel perspective on maintaining orthogonality efficiently. Rather than enforcing exact orthogonality at every optimization step, the authors amortize the computational cost through drift-triggered restoration. This design is intuitive and provides a reasonable way to balance computational efficiency and orthogonality preservation.

  3. The paper provides solid theoretical analysis and proofs that offer a rigorous foundation for the proposed update and restoration mechanism.

  4. The experiments are well designed and cover a broad range of models, tasks, and orthogonal parameterizations. The results are promising, demonstrating improved computational efficiency while maintaining or improving downstream performance. The extensive ablation studies further strengthen the empirical support for RTOFT.

Critical weaknesses
  1. Limited sensitivity analysis of τ\tau. The choice of drift tolerance τ\tau is crucial to RTOFT. Although the paper provides some sensitivity analysis, it is mainly limited to CoLA and RTE. A more systematic evaluation on larger models and additional tasks would help clarify how different values of τ\tau jointly affect downstream accuracy, restoration frequency, and actual training time.

  2. Insufficient investigation of block size bb. RTOFT adopts the block-diagonal parameterization of existing OFT methods, with block sizes following previous settings. However, since RTOFT changes the optimization procedure and substantially reduces per-step orthogonal construction costs, the block sizes used in conventional OFT may not necessarily remain optimal. The paper lacks a systematic analysis of how bb affects downstream performance and computational efficiency. Since RTOFT reduces the per-step computational overhead of orthogonal construction, could increasing the block size bb improve adaptation capacity with only a modest increase in training cost?

Questions for authors
  1. The authors observe that the orthogonality defects of OFTv2 span five orders of magnitude. What are the practical implications of such variations? In particular, do these differences in orthogonality defects have a measurable impact on downstream performance or training stability?

  2. The paper demonstrates that the truncation order in OFTv2 does not reliably control realized orthogonality and explains this through the dependence on the learned parameter norm and truncation order. Could the authors further clarify the practical implications of this finding? In particular, could a carefully selected truncation order provide a simpler alternative to RTOFT?

  3. Previous OFT variants, including OFT, COFT, BOFT, and PSOFT, suggest that relaxing strict orthogonality constraints may sometimes improve downstream adaptation. Does this indicate that some degree of flexibility in orthogonality preservation is beneficial for downstream fine-tuning? How does RTOFT balance orthogonality preservation and model plasticity, particularly when choosing the drift tolerance?

Reviewer_JgaNOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Orthogonal fine-tuning [1] adapts pretrained weights by multiplying them by an orthogonal matrix RR, giving a parametrization of WRWR. Existing methods construct an exact or approximate orthogonal RR at every step. This paper introduces Restarted Tangent Orthogonal Finetuning (RTOFT), which instead parameterizes the updates as W(I+S)W(I + S), where SS is a skew-symmetric matrix. The authors measure the drift from orthogonality throughout training and, once it crosses a predefined threshold, restore orthogonality via a polar decomposition that projects I+SI + S onto Od(R)\mathrm{O}_d(\mathrm{R}). The identity ∣∣(I+S)⊤(I+S)−I∣∣2=∣∣S∣∣22||(I + S)^\top(I + S) - I||_2 = ||S||_2^2 allows to measure the orthogonality defect efficiently. Experiments on language and image models show that RTOFT matches or improves the task performance of existing orthogonal fine-tuning methods while reducing training time to roughly that of low-rank fine-tuning. In the retention experiments, it preserves general capabilities at least as well as the matched orthogonal baseline. Removing restoration causes more forgetting.

Critical strengths
  • (S1) Simplicity and interpretablity. The main appeal for this method in my opinion is using the measured drift to decide when orthogonality needs to be restored. The orthogonality defect gives the tolerance a geometric meaning and a guarantee that doesn't depend on the loss or optimizer.

  • (S2) Evidence for restoration during training. The ablations show that restoration improves task performance and retention. The comparisons with per-step restoration also support using the cheaper drift-triggered schedule..

Critical weaknesses
  • (W1) Incomplete evidence for every-step monitoring. On PEFT-Arena [2], drift is checked only every twenty steps, and restoration happens at almost every check. This doesn't establish the guarantee for every forward pass. I'd want to see if there are benefits to checking every step.

  • (W2) Speedup may be inflated by evaluation overhead. OFTv2 [3] rebuilds its transform for every evaluation batch in the runtime comparison. Since its parameters stay fixed during evaluation, the transform could be computed once and reused. I'd want to see whether the reported speedup holds after removing this overhead from the baseline.

Questions for authors
  • (Q1) Optimized baseline comparison. How much end-to-end speedup remains against OFTv2 with the transform cached during evaluation?

  • (Q2) Every-step monitoring. Does checking drift after every optimizer step on PEFT-Arena preserve change the results? What runtime overhead does it add?


[1] Qiu et al. Controlling text-to-image diffusion by orthogonal finetuning. NeurIPS, 2023. arXiv:2306.07280.

[2] Huang et al. PEFT-Arena: Understanding parameter-efficient finetuning from a stability-plasticity perspective. arXiv:2605.28819, 2026.

[3] Qiu et al. Orthogonal finetuning made scalable. EMNLP, 2025. arXiv:2506.19847.

Reviewer_iwBBOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces RTOFT, a more computationally efficient way of performing orthogonal fine-tuning. Standard OFT construct an orthogonal transformation at every optimisation step. The paper argues that this is not necessary, and RTOFT takes cheaper tangent-space updates: Wk(I+S),ST=−SW_k(I+S), S^T=-S. This allows the intermediate transformation to be measurably non-orthogonal. The method restores exact orthogonality using polar(I+S)polar(I+S). Orthogonality can therefore be amortised over multiple steps.

Empirically, the method reduces the substantial training-time overhead of OFT methods: the authors report conventional orthogonal methods at roughly 1.10–1.58×1.10–1.58\times LoRA wall-clock and restarted versions at 0.97–1.12×0.97–1.12\times across models, tasks and modalities.

Critical strengths
  • The core idea of the paper is clean and well-motivated. The authors provide a way of amortising orthogonality over multiple training steps with a controllable restart algorithm.
  • The paper contains a strong set of controlled ablations. It distinguishes tangent adaptation without restoration, fixed-period restoration, drift-triggered restoration, per-step restoration, and different inner maps, and it evaluates the approach across multiple models, modalities, and orthogonal parameterisations.
  • The implementation details are considered. In particular, the authors explicitly account for the numerical precision required by the polar restoration and provide operation-level and wall-clock analyses of the restoration overhead.
Critical weaknesses
  • The evaluation shows certain suspicious numbers that I need deeper understanding, specifically:
    • The full-FT mostly underperforms the low-rank and OFT solutions. Why is this the case, could the LR be too large for that setting? Could the full-FT baseline be undertuned?
    • The latency of full-FT is also singificantly lower than the PEFT counterparts.
    • The RTFOT improves accuracy over OFTv2.
  • Much of the GLUE gain is not from the novel drift trigger. Efficiency results are based primarily on fixed-period restoration. Demonstrating drift-triggered experiments across all different parametrisations would strengthen this central claim.
  • Most detailed runtime evidence is on relatively structured/small adapters. Monitoring/restoration overhead at substantially larger model scale (e.g., 30-70B scale) and in distributed training remains unmeasured.
Questions for authors
  • RTOFT doesn't merely retain OFT quality while becoming faster, but sometimes it substantially improves it. Why is this the case? This would require deeper investigation. For example, GLUE goes: $ 88.81 \text{(OFTv2)} \rightarrow 89.90 \text{(RTOFT)}. $ And on Qwen commonsense: $ 86.82 \rightarrow 88.38 $. What mechanism do the authors believe explains the remaining performance improvement?
  • The Appendix provides a useful period τ\tau sweep on CoLA and RTE. How sensitive is the quality–cost trade-off to τ\tau on the larger Qwen and Stable Diffusion experiments?
  • Can the trigger for restoration be layer dependent or does it need to be global?
  • Why do the Stable Diffusion comparison use very different parameter budgets?
  • What happens under tensor parallelism or distributed training?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.