- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes Fixed-Reference Tuning (FRET), a parameter-efficient fine-tuning method combining orthogonal transformations of pretrained weights (OFT) with normalized low-rank adaptation (NoRA). The authors observe that normalizing the adapter matrix A before composing it with a learned rotation does not preserve the effective projection’s column norms in the original input coordinates. FRET instead enforces normalization in those fixed coordinates. The paper introduces two variants: FRET-C normalizes the combined projection AR^T while FRET-D learns an independent normalized projection alongside the rotated pretrained weights. The authors provide a mathematical analysis of how rotations redistribute column norms, examine the variants’ optimization dynamics, and show that the adapted layers can be merged for inference. Experiments on six mathematical-reasoning benchmarks report improvements of 0.43 and 0.31 percentage points over the naive combination for FRET-C and FRET-D, respectively.
Critical strengths
-
Simple design supported by mathematical analysis: The paper identifies a concrete interaction between normalization and learned rotations and proposes a straightforward modification to control the effective projection norms.
-
Targeted ablation with consistent gains: The experiments directly compare the proposed normalization placement against the naive combination. FRET-C improves performance across all three paired training seeds, averaging 0.43 percentage points. Although the gains are modest and the evaluation is limited, their consistency supports the usefulness of the modification within the tested setting.
Critical weaknesses
-
Limited methodological novelty and incomplete discussion of prior work: The main methodological change is the placement of normalization within a combination of existing successful adaptation techniques. Although FRET introduces a specific normalization constraint and analyzes its properties, the proposed normalization placement appears to be a straightforward adjustment to the combination of OFT and NoRA, requiring limited additional methodological insight. I therefore find the advance incremental, and the paper does not sufficiently establish its broader significance. Furthermore, combining OFT with LoRA has already been explicitly investigated by [1], but this relevant work is not discussed.
-
Limited experimental support for the benefit and its explanation: The improvements are modest and evaluated only on mathematical-reasoning benchmarks under one shared training recipe. Also, the paper does not measure how much rotations alter the effective projection’s column norms in trained models or test whether the original input coordinates are preferable to another fixed reference basis. Consequently, the experiments provide limited evidence that the proposed mechanism explains the improvement or that the benefit generalizes beyond the limited tested setting.
[1] Tao et al. Transformed Low-rank Adaptation via Tensor Decomposition and Its Applications to Text-to-image Models. ICCV, 2025.
Questions for authors
-
Methodological significance (clarification): What makes FRET a nontrivial methodological advance over a straightforward combination of OFT and NoRA? In particular, what is the key insight behind the proposed normalization placement, and what establishes its significance? Please also clarify the relationship to the prior OFT+LoRA combination in [1].
-
Experimental setup and supporting evidence (clarification): a. Given the modest improvements and evaluation limited to mathematical reasoning, what evidence supports FRET’s broader usefulness? Do you have additional results on other task types, models, or training settings that establish the robustness of these gains? b. Have you measured the changes in the effective projection’s column norms in the trained naive baseline, and do you have evidence that the original input coordinates are preferable to another fixed reference basis? Please distinguish existing evidence from hypotheses that remain untested.