acceptodds
Under review as a conference paper at ICLR 2027

SpectraGain: Rank-Decoupled Weight Adaptation via Low-Rank Spectral Gains

Abstract

Directly factorizing a weight correction ties its rank to the trainable-parameter budget. We introduce Carrier-Conditioned Spectral Gain Adaptation (SpectraGain), which instead factorizes a real-valued spectral gain and applies it to the frozen weight spectrum. This produces a carrier-conditioned circular update whose realized rank can exceed the generator rank, while preserving the pretrained model at initialization. We characterize the resulting update structure and evaluate SpectraGain across language, reasoning, and vision tasks. Controlled comparisons with closely related operators, same-generator coupling replacements, and fixed-checkpoint rank interventions examine how coupling geometry and correction structure shape downstream performance. A rematerialized implementation further enables long-context training, with its memory and runtime costs characterized against standard and fused low-rank adaptation.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
parameter-efficient fine-tuning, transfer learning, low-rank adaptation, spectral adaptation, weight-space adaptation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_AfgAOpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes SpectraGain, a PEFT operator that learns a compact spectral gain and applies it to the Fourier representation of a frozen weight matrix. The resulting carrier-conditioned circular correction can have a substantially higher realized rank than the compact trainable generator.

The paper evaluates the method across language and vision tasks and compares it with low-rank, high-rank, Fourier-based, and circular adaptation methods. It also includes same-generator circular-versus-pointwise comparisons, tangent-space analyses, correction-rank interventions, and runtime and memory measurements.

Critical strengths
  1. The paper provides a technically informative analysis of different adaptation geometries, especially the distinction between generator rank and realized correction rank.

  2. The controlled same-generator experiments and systems analyses are careful and strengthen the scientific value of the work.

Critical weaknesses
  1. The conceptual novelty is somewhat compositional. Closely related prior work already studies spectral weight adaptation, carrier-conditioned pointwise modulation, and additive circular adaptation separately. SpectraGain’s main novelty is the specific combination of a low-rank spectral gain with frozen-carrier conditioning and circular coupling, rather than introducing each of these ingredients independently.

  2. The controlled experiments support a task-dependent effect of circular versus pointwise coupling rather than a consistent advantage of SpectraGain. Circular coupling is favorable on T5 but unfavorable on CLIP in the fixed-checkpoint comparison, while the independently retrained differences are small and statistically inconclusive.

  3. The practical efficiency benefit is strongly sequence-length dependent. At 1K tokens, SpectraGain takes 2.064 seconds per step compared with 0.241 seconds for fused LoRA, while the relative overhead becomes much smaller at long sequence lengths. The efficiency case is therefore most compelling for long-context training rather than for PEFT workloads in general.

Questions for authors

Given that the circular-versus-pointwise effect changes sign between T5 and CLIP, what is the intended general conclusion: that circular coupling is expected to be preferable under identifiable task conditions, or that the main contribution is to characterize task-dependent coupling geometry rather than to establish a generally superior coupling operator?

Reviewer_beS5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SpectraGain, a parameter-efficient fine-tuning method that factorizes a real-valued spectral gain rather than directly factorizing the weight update. The generated gain modulates the Fourier spectrum of the frozen weight matrix, allowing a low-rank generator to induce a potentially high-rank weight correction while exactly preserving the pretrained model at initialization. The authors evaluate SpectraGain on language understanding, commonsense reasoning, instruction following, code generation, and image classification using T5, LLaMA, and CLIP. The paper also studies circular versus pointwise coupling, post-hoc rank truncation of learned corrections, and the memory and runtime costs of long-context training.

Critical strengths
  1. The proposed parameterization is clear and reasonably novel. SpectraGain applies low-rank factorization to a spectral gain rather than directly to the weight correction, which distinguishes it from LoRA, HiRA, and FourierFT.

  2. The empirical evaluation is broad. The paper covers language, reasoning, code generation, and vision tasks while approximately matching the trainable-parameter budgets of the compared methods. The improvement over HiRA on Commonsense-15K is particularly encouraging.

Critical weaknesses
  1. My main concern is whether circular coupling provides a meaningful inductive bias for weight matrices. Hidden-channel orderings are generally arbitrary up to function-preserving permutations, whereas SpectraGain treats neighboring indices and wraparound boundaries as structurally related. The paper does not test whether its performance depends on the particular channel ordering of the pretrained checkpoint.

  2. Except for the Commonsense-15K comparison, the improvements over the closest baselines are generally small. SpectraGain obtains 87.62 versus 87.59 for HiRA on T5 and 90.88 versus 90.76 on CLIP. Moreover, the confidence intervals for the independently retrained same-generator comparisons include zero. The current evidence supports the claim that SpectraGain is competitive, but not that circular coupling is generally superior to pointwise coupling.

Questions for authors
  1. How sensitive is SpectraGain to function-preserving permutations of the hidden channels? A comparison among the original ordering and several randomly permuted but functionally equivalent checkpoints would help determine whether the circular neighborhood captures meaningful pretrained structure or an arbitrary coordinate ordering.

  2. Why does the 8r truncated correction in Figure3 slightly outperform the full correction? Could the authors report paired per-task and per-seed differences among the 2r, 4r, 8r, and full conditions, together with appropriate uncertainty estimates?

Reviewer_UNCKOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SpectraGain, a fine-tuning method that learns low-rank gains on the Fourier spectrum of the frozen weights to produce a circulant update. It is well analyzed and beats HiRA by about one point on Commonsense-15K. However, the novelty is narrow, gains on other tasks are small, and the cost–accuracy trade-off is unclear.

Critical strengths
  1. The adaptation operator is clearly defined. The method learns low-rank multiplicative gains on the Fourier spectrum of the frozen weights, and then applies the inverse transform to get the update (Table 1, Eqs. 1–5). This differs from elementwise scaling in the original weight space, and also from learning an additive residual in the Fourier domain. The contribution is a family of circulant updates that depend on the original weights.
  2. Expressivity and implementation are explained in detail. The paper separates the rank of the generator from the rank of the final update. It also handles the Hermitian symmetry needed for a real-valued inverse transform. In addition, it explains the phase limits of real-valued gains, the identity initialization, and how the weights are merged. This level of detail makes the method easy to analyze precisely.
  3. The comparison with the closest baseline is direct and well documented. With the same target layers and roughly the same number of parameters, SpectraGain gets 82.69 on Commonsense-15K and HiRA gets 81.68 (Table 3). Over three paired seeds, the mean gain is 1.01 points, with a reported 95% CI of [0.850, 1.171]. The tables mark which results are cited and which are newly run. For T5 and CLIP, hyperparameters are chosen on a dev split taken from the training set, and the gradient diagnostics are fixed before final evaluation (Appendix D). This supports the claim that the method is useful on at least some tasks.
  4. The analysis of where the gains come from is careful. The HiRA, C3A-style, and same-generator comparisons each test a different structural factor. Retraining from scratch, swapping at a fixed checkpoint, and truncating the update rank each support a different kind of conclusion. The paper openly reports task dependence, counterexamples, and intervals that include zero. The long-sequence experiments include fused LoRA, which separates the savings from recomputation from the extra cost of the spectral operator.
Critical weaknesses
  1. The novelty is limited to one specific operator. Compared with existing high-rank, frequency-domain, and circulant methods, the new part is mainly the spectral gain that depends on the frozen weights, together with circulant coupling. The authors also note in Appendix A.2 that elementwise multiplication can already separate generator rank from update rank. Being able to reach high rank does not by itself show better expressivity on real tasks.
  2. The gains show up only on some tasks. The average gain over HiRA is 0.03 points on T5 and about 0.11 points on CLIP. When the same generator is retrained from scratch (Section 4.5), the circulant-minus-elementwise difference is +0.107 points on T5 and −0.058 points on CLIP, and both confidence intervals include zero. The authors already describe this as a task-dependent coupling effect. Still, the average gains are small, and there is little evidence that the results carry over to other tasks. Also, comparisons with cited baselines depend on whether the original protocols match. The newly run comparisons at similar budgets are more reliable.
  3. The cost–accuracy trade-off in the main setting is unclear. The paper is open about the FFT overhead on short sequences. With rank one, LLaMA2-7B, and 1K tokens, SpectraGain takes 2.064 s/step and fused LoRA takes 0.241 s/step, about 8.6 times slower. The relative overhead shrinks for longer sequences: at 96K tokens, the extra time is 0.864 s/step and the extra memory is 0.94 GiB. However, the cost tests use a different backbone and rank from the main accuracy experiments. So it is hard to judge the actual trade-off between accuracy, time, and memory.
Questions for authors
  1. Can you report accuracy, training time, and peak memory at the ranks and target layers of the main accuracy experiments, against fused LoRA under the same conditions?
  2. On one representative task, can you build an additive update from the same generator's spectral kernel, with matched initialization scale, to isolate the effect of using the frozen weights as the carrier?
  3. As an optional check: if the selection rule is fixed in advance, does choosing between circulant and elementwise updates still help on held-out tasks or new initializations?
1 comment

Thanks to the authors for their response and additional experiments. These are helpful, though my concerns about the limited methodological novelty and the small gains over HiRA on T5 and CLIP remain. I think further refinements to the method and improvements to the experiments could make the paper more convincing. I will therefore keep my original score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.