acceptodds
Under review as a conference paper at ICLR 2027

FRET: FIXED-REFERENCE TUNING

Abstract

Low-rank adaptation learns through a compact projection of the model's input features. Normalizing this projection controls the contribution of each input channel, yet a learned rotation redistributes these gains through correlations between projection columns. We introduce Fixed-Reference Tuning (FRET), which imposes unit column norms on the effective projection in the layer's input coordinates. Its coupled realization projects the raw factor onto this constraint at each forward pass; its decoupled realization learns the constrained projection through independent coordinates. We characterize gain redistribution and show that coupled normalization removes the radial component of a rotation update at identity. The resulting layer combines a rotated frozen base with a normalized low-rank residual and admits exact dense export. Across six mathematical-reasoning benchmarks and three paired training seeds, both realizations exceed the naive moving-frame combination: FRET-C by macro points, with no negative benchmark–seed difference, while the two realizations differ by points, within seed variation.

open until 14 Dec 2026

est. 32% chance this paper gets accepted at ICLR 2027.

Reject 68%Accept 32%

What do you think this paper will get?

All positions stay anonymous.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.