- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The authors investigate whether adaptation under parameter-efficient fine-tuning (PEFT) must occur along specific spectral directions by evaluating leading, middle, tail, and random spectral supports, both with and without rotation.
To examine the role of interactions between leading and tail directions, they also compare training with interaction penalty regularization, no regularization, and norm-matched regularization.
Finally, based on the results of removing cross-region components from the learned parameter updates, they conclude that cross-region structure plays an important role in adaptation.
Critical strengths
The authors identify a limitation of prior methods such as PiSSA and MiLoRA, which set the adaptation directions only at initialization, and design experiments that can confine the adaptation directions throughout training.
This allows them to conduct controlled experiments examining the importance of principal directions during the training process.
Critical weaknesses
1. The analytical contribution is limited.
The authors explicitly acknowledge that several results are statistically unresolved, limiting the conclusions that can be drawn about the importance of spectral directions or their interactions.
Furthermore, several findings are difficult to interpret, such as the discrepancy between the trends in NLL and accuracy, and the observation that MIX reduces NLL but loses this advantage after temperature scaling.
As a result, the analysis does not seem to provide readers with sufficiently clear intuition about the underlying mechanisms.
2. The writing lacks clarity and polish.
Several terms are used without proper definitions, such as coefficient/rotated adapter in the Introduction, references to encoder and decoder without specifying the models or tasks being considered, and UNREG/MIX/NORM appearing in Section 3 before being defined.
Additionally, terms such as support, band, and region do not appear to be used consistently, and the transitions between sentences are often awkward.
The motivations behind the experiments and the interpretations of their results also do not seem to be presented in a sufficiently coherent and well-structured manner.
Questions for authors
1. Could you provide results on more recent model families, such as Qwen3, and a broader range of tasks, even if the model sizes remain similar to those currently evaluated?
The current selection of model families and tasks seems insufficient to support the conclusion that principal directions are not necessary for successful adaptation.
2. The interaction experiments rely on regularization as an indirect way of controlling interactions, rather than explicitly preventing the interaction components from being learned.
Would it not be possible to directly disable these components by formulating the parameter update as and fixing the cross-region blocks and to zero during training?
I would like to understand why the interaction experiments need to retain the original formulation based on and .
3. In Section 5.2, you compare completely removing the cross-region components after training with uniformly shrinking the parameter update to match its norm.
However, uniform shrinking preserves the direction of the parameter update, whereas removing the cross-region components can substantially alter its direction, potentially making the comparison unfavorable to cross-region removal.
A fairer comparison might be to compare removing the cross-region components with removing an equal amount of norm from the and components.
Could you provide results from such an experiment?
2 comments
Thank you for the detailed response. After carefully considering the rebuttal, I will maintain my original score.
W1. Analytical contribution and mechanisms
With every band tuned, all bands improve accuracy and none has a measurable advantage, while LoRA, which learns its directions, reaches lower NLL than every tuned band (general comment, "Adapter tuning based on learning rates" and "Narrower supports"; Appendix C.3).
We agree the NLL trends in the submission were hard to interpret. They do not persist with tuned learning rates and early stopping, and the revision no longer claims them (Appendices C.1 and C.2).
On the mechanism, tuning appears to compensate for spectral scale: tail coefficients grow to 0.8–6.4 times their singular values, against 0.07–0.15 for leading ones, at comparable accuracy (Appendix C.5).
W2. Clarity
Agreed. The Introduction now defines band, support, the C and R adapters and UNREG/MIX/NORM. The revision uses band in place of region throughout, names models and tasks in the main text instead of "encoder" and "decoder", and leads Section 5.1 with the tuned results (Table 1).
Q1. Qwen3 and more tasks
Yes. We added Qwen3-1.7B on CommonsenseQA and a second task set (ARC-Easy, ARC-Challenge and OpenBookQA) on Qwen2.5-3B (general comment, "Adapter tuning based on learning rates"; Table 1). In both, every band improves accuracy and none has a measurable advantage, so we find no evidence that principal directions are required.
Q2. Why not fix the cross blocks to zero?
The suggested form makes every block a free matrix: even with the cross blocks fixed to zero it trains k² + r² parameters per matrix, about 1.3 million on Qwen2.5-1.5B (d = 1536, r = 512). Ours trains r + 2d, about 3.6 thousand (Eq. 7, Appendix D.1).
We designed the adapter to stay linear in the layer width while still allowing limited interaction. Its diagonal scalers create that interaction, which separates exactly into cross-band blocks for analysis (Eq. 10). These blocks are not parameters, so they cannot be fixed to zero without removing the scalers (Appendix A.7), and we control them with a penalty (Section 3.2).
Q3. Removing an equal amount from the within-band blocks
We ran this control (Section 5.2, Table 3). Removing the cross blocks raises NLL 1.6–3.2 times as much as removing the same norm from C_LL and C_TT, while accuracy differs by less than 0.6 points. The trained model thus makes some use of the cross-band components, but accuracy does not depend on them, as the random-partition control also shows (general comment, "Further control: leading–tail interaction").