acceptodds
Withdrawn from ICLR 2027

FRET: FIXED-REFERENCE TUNING

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Low-rank adaptation learns through a compact projection of the model's input features. Normalizing this projection controls the contribution of each input channel, yet a learned rotation redistributes these gains through correlations between projection columns. We introduce Fixed-Reference Tuning (FRET), which imposes unit column norms on the effective projection in the layer's input coordinates. Its coupled realization projects the raw factor onto this constraint at each forward pass; its decoupled realization learns the constrained projection through independent coordinates. We characterize gain redistribution and show that coupled normalization removes the radial component of a rotation update at identity. The resulting layer combines a rotated frozen base with a normalized low-rank residual and admits exact dense export. Across six mathematical-reasoning benchmarks and three paired training seeds, both realizations exceed the naive moving-frame combination: FRET-C by macro points, with no negative benchmark–seed difference, while the two realizations differ by points, within seed variation.

Primary area:
foundation or frontier models, including LLMs
Keywords:
parameter-efficient finetuning, low-rank adaptation, orthogonal finetuning, RL post-training

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_qWD6OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Fixed-Reference Tuning (FRET), a parameter-efficient fine-tuning method combining orthogonal transformations of pretrained weights (OFT) with normalized low-rank adaptation (NoRA). The authors observe that normalizing the adapter matrix A before composing it with a learned rotation does not preserve the effective projection’s column norms in the original input coordinates. FRET instead enforces normalization in those fixed coordinates. The paper introduces two variants: FRET-C normalizes the combined projection AR^T while FRET-D learns an independent normalized projection alongside the rotated pretrained weights. The authors provide a mathematical analysis of how rotations redistribute column norms, examine the variants’ optimization dynamics, and show that the adapted layers can be merged for inference. Experiments on six mathematical-reasoning benchmarks report improvements of 0.43 and 0.31 percentage points over the naive combination for FRET-C and FRET-D, respectively.

Critical strengths
  1. Simple design supported by mathematical analysis: The paper identifies a concrete interaction between normalization and learned rotations and proposes a straightforward modification to control the effective projection norms.

  2. Targeted ablation with consistent gains: The experiments directly compare the proposed normalization placement against the naive combination. FRET-C improves performance across all three paired training seeds, averaging 0.43 percentage points. Although the gains are modest and the evaluation is limited, their consistency supports the usefulness of the modification within the tested setting.

Critical weaknesses
  1. Limited methodological novelty and incomplete discussion of prior work: The main methodological change is the placement of normalization within a combination of existing successful adaptation techniques. Although FRET introduces a specific normalization constraint and analyzes its properties, the proposed normalization placement appears to be a straightforward adjustment to the combination of OFT and NoRA, requiring limited additional methodological insight. I therefore find the advance incremental, and the paper does not sufficiently establish its broader significance. Furthermore, combining OFT with LoRA has already been explicitly investigated by [1], but this relevant work is not discussed.

  2. Limited experimental support for the benefit and its explanation: The improvements are modest and evaluated only on mathematical-reasoning benchmarks under one shared training recipe. Also, the paper does not measure how much rotations alter the effective projection’s column norms in trained models or test whether the original input coordinates are preferable to another fixed reference basis. Consequently, the experiments provide limited evidence that the proposed mechanism explains the improvement or that the benefit generalizes beyond the limited tested setting.

[1] Tao et al. Transformed Low-rank Adaptation via Tensor Decomposition and Its Applications to Text-to-image Models. ICCV, 2025.

Questions for authors
  1. Methodological significance (clarification): What makes FRET a nontrivial methodological advance over a straightforward combination of OFT and NoRA? In particular, what is the key insight behind the proposed normalization placement, and what establishes its significance? Please also clarify the relationship to the prior OFT+LoRA combination in [1].

  2. Experimental setup and supporting evidence (clarification): a. Given the modest improvements and evaluation limited to mathematical reasoning, what evidence supports FRET’s broader usefulness? Do you have additional results on other task types, models, or training settings that establish the robustness of these gains? b. Have you measured the changes in the effective projection’s column norms in the trained naive baseline, and do you have evidence that the original input coordinates are preferable to another fixed reference basis? Please distinguish existing evidence from hypotheses that remain untested.

Reviewer_pUFsOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Fixed-Reference Tuning, a novel parameter-efficient fine-tuning method designed to address the "gain drift" issue in low-rank adaptations combined with orthogonal rotations. The authors observe that applying normalization in a moving frame causes first-order radial gain changes, altering the magnitude of the effective projection. FRET instead enforces a unit-gain constraint in the fixed, original input coordinate space. The authors provide a rigorous mathematical characterization of this constraint, deriving a nearest-feasible factor solution and an equivalent coordinate parameterization. Theoretically, the authors prove that FRET removes the radial component responsible for gain drift while retaining tangential directional updates. On six mathematical reasoning benchmarks using three paired random seeds, FRET consistently outperforms the moving-frame normalization baseline and demonstrates high stability.

Critical strengths
  1. Rigorous Theory: The mathematical characterization of the gain drift is excellent. The authors clearly demonstrate why moving-frame normalization fails to preserve unit gain and how the fixed-reference constraint elegantly projects the update onto the tangent space of the unit sphere, thereby preserving the radial component.
  2. Systematic Methodological Design: The distinction between FRET-C and FRET-D , and the proof that they span the exact same function class while behaving differently under optimizers, is a strong and valuable contribution to the PEFT optimization literature.
  3. Rigorous Experimental Setup: The comparison uses paired seeds and evaluates on six challenging mathematical reasoning benchmarks. The high consistency of the results strongly supports the claim that the placement of the constraint is beneficial, even if the absolute margin is small.
Critical weaknesses
  1. Incomplete Mechanism Verification: Section 4.3 explicitly states that within-block column correlations, rotation displacement, and realized gain errors were not recorded for the trained layers. Without these measurements, the causal link between the theoretical mechanism and the empirical performance improvement remains unverified in the actual models.
  2. Limited Task Scope: The experiments are exclusively limited to mathematical reasoning benchmarks. It is unclear how FRET performs on standard natural language processing tasks or multimodal tasks, where the input feature distributions and correlation structures might differ significantly.
  3. Complexity and Hyperparameter Overhead: The introduction of the reset mechanism, the choice between FRET-C and FRET-D, and the block rotation width add significant implementation and hyperparameter tuning overhead compared to standard LoRA or DoRA.
Questions for authors
  1. Regarding Section 4.3, could you provide the actual checkpoint measurements for the realized gain error and rotation displacement? Demonstrating that FRET actually reduces this error in trained models would strongly support your theoretical claims.
  2. Have you evaluated FRET on other standard PEFT benchmarks (e.g., natural language understanding or generation)? Does the fixed-reference constraint hold the same advantage when input feature channel correlations are low?
  3. Computational Overhead: Could you provide explicit wall-clock training time and inference latency comparisons between FRET, Naive, and standard LoRA?
Reviewer_EYGvOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Fixed-Reference Tuning (FRET), which combines learned orthogonal rotations with normalized low-rank adaptation. It introduces coupled and decoupled variants to preserve input-coordinate projection norms and provides theoretical analyses of gain redistribution. Experiments on six mathematical reasoning benchmarks report modest improvements over moving-frame normalization.

Critical strengths

The paper presents a mathematically motivated formulation of fixed-reference normalization and provides theoretical analyses of its interaction with learned rotations. The method also supports merging into a dense weight matrix for inference.

Critical weaknesses

1.Although the paper theoretically characterizes rotation-induced gain redistribution, it does not demonstrate that this phenomenon is significant or detrimental in actual model training. The experiments report only a marginal performance improvement without measuring the proposed mechanism in trained checkpoints. Furthermore, the lack of a random fixed-reference baseline leaves the necessity of the original input coordinate system unverified. 2.The paper lacks concrete implementation details, including the exact base model, training data, and key hyperparameters, making the experiments difficult to reproduce. Furthermore, Figures 1 and 2 are embedded as raster images rather than vector graphics, which reduces their presentation quality.

Questions for authors

Please refer to the concerns raised in the Critical Weaknesses section.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.