- Rating
- 3
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
OFT preserves the geometric properties of pretrained models, but existing methods require exact or approximate orthogonal transformations at every optimization step, introducing additional computational overhead. The authors observe that the approximation scheme used in OFTv2 does not reliably control the realized orthogonality. To address this issue, they propose RTOFT, which replaces per-step orthogonal construction with inexpensive tangent updates. RTOFT restores exact orthogonality only when the measured drift reaches the predefined threshold , thereby amortizing the cost of maintaining orthogonality across optimization steps. Extensive experiments across multiple datasets and models demonstrate that RTOFT achieves competitive or improved downstream performance while substantially reducing the computational overhead of OFT.
Critical strengths
-
The paper is well motivated and addresses a genuine challenge in OFT. The additional computational overhead required to maintain orthogonality is a key limitation of existing OFT methods, making this an important and practically relevant problem.
-
RTOFT offers a novel perspective on maintaining orthogonality efficiently. Rather than enforcing exact orthogonality at every optimization step, the authors amortize the computational cost through drift-triggered restoration. This design is intuitive and provides a reasonable way to balance computational efficiency and orthogonality preservation.
-
The paper provides solid theoretical analysis and proofs that offer a rigorous foundation for the proposed update and restoration mechanism.
-
The experiments are well designed and cover a broad range of models, tasks, and orthogonal parameterizations. The results are promising, demonstrating improved computational efficiency while maintaining or improving downstream performance. The extensive ablation studies further strengthen the empirical support for RTOFT.
Critical weaknesses
-
Limited sensitivity analysis of . The choice of drift tolerance is crucial to RTOFT. Although the paper provides some sensitivity analysis, it is mainly limited to CoLA and RTE. A more systematic evaluation on larger models and additional tasks would help clarify how different values of jointly affect downstream accuracy, restoration frequency, and actual training time.
-
Insufficient investigation of block size . RTOFT adopts the block-diagonal parameterization of existing OFT methods, with block sizes following previous settings. However, since RTOFT changes the optimization procedure and substantially reduces per-step orthogonal construction costs, the block sizes used in conventional OFT may not necessarily remain optimal. The paper lacks a systematic analysis of how affects downstream performance and computational efficiency. Since RTOFT reduces the per-step computational overhead of orthogonal construction, could increasing the block size improve adaptation capacity with only a modest increase in training cost?
Questions for authors
-
The authors observe that the orthogonality defects of OFTv2 span five orders of magnitude. What are the practical implications of such variations? In particular, do these differences in orthogonality defects have a measurable impact on downstream performance or training stability?
-
The paper demonstrates that the truncation order in OFTv2 does not reliably control realized orthogonality and explains this through the dependence on the learned parameter norm and truncation order. Could the authors further clarify the practical implications of this finding? In particular, could a carefully selected truncation order provide a simpler alternative to RTOFT?
-
Previous OFT variants, including OFT, COFT, BOFT, and PSOFT, suggest that relaxing strict orthogonality constraints may sometimes improve downstream adaptation. Does this indicate that some degree of flexibility in orthogonality preservation is beneficial for downstream fine-tuning? How does RTOFT balance orthogonality preservation and model plasticity, particularly when choosing the drift tolerance?