- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper investigates which parameter subspaces remain stable during finetuning. Through experiments across vision and language models, the authors find that leading spectral subspaces are largely preserved, particularly in shallower layers, and that their stability increases with pretraining data scale. Based on these observations, the authors show that downstream adaptation can be restricted to the leading spectral subspaces of deeper layers while maintaining competitive performance. The authors claim that these findings provide a parameter-space perspective on how pretraining shapes downstream adaptation.
Critical strengths
1.Understanding the spectral relationship between pretrained and finetuned models remains an open problem, and the paper offers an interesting perspective by investigating stable rather than changing parameter subspaces.
2.The empirical analysis is relatively comprehensive. The authors investigate spectral stability across different architectures, layers, and tasks, supported by cross-task singular-vector swapping, pretraining-scale experiments, and analyses of spectral rank and layer selection. These experiments provide useful empirical insights.
Critical weaknesses
1.The proposed adaptation closely resembles LoRA-XS [1], which also freezes SVD-derived projections and trains only an intermediate matrix. However, LoRA-XS is neither cited nor discussed. The paper also insufficiently positions its findings relative to PiSSA, SVFT, and MiLoRA [2], raising concerns about the novelty of its methodological insights.
2.Section 3.2 does not establish a convincing link between the observed spectral stability and the proposed adaptation method. The authors find that leading singular subspaces remain stable during finetuning, but they subsequently restrict adaptation to these same subspaces. Stability alone does not establish that these are the most effective update directions. The proposed method therefore lacks sufficient justification from the preceding analysis.
3.Although the paper examines multiple vision and language architectures, its key functional and cross-task evaluations focus on encoder-based language models and GLUE tasks, while controlled pretraining experiments use a small-scale vision model. The absence of modern decoder-only LLMs and more challenging downstream tasks limits the generalizability of the conclusions.
[1] Bałazy et al. LoRA-XS: Low-Rank Adaptation with Extremely Small Number of Parameters. ArXiv 2024.
[2] Wang et al. MiLoRA: Harnessing Minor Singular Components for Parameter-Efficient LLM Finetuning. NAACL 2025.
Questions for authors
1.How is the proposed adaptation method motivated by the observed spectral stability? Why should adaptation target stable leading subspaces rather than directions that change more substantially?
2.How does the proposed method differ from LoRA-XS? Given their effectively equivalent update spaces, what distinct methodological insight does the proposed parameterization provide, and why is LoRA-XS not discussed?
3.How well do the findings generalize to modern LLMs and more challenging tasks? Additional evidence from decoder-only LLMs and more diverse downstream adaptation settings would strengthen the paper's broader claims.
4 comments
Response (Part 2)
Questions
Q1. Justification for Leading-Subspace Adaptation
Our adaptation strategy is motivated by hierarchical analyses of spectral stability at both the layer and rank levels. At the layer level, shallow layers exhibit small changes across their spectral components during fine-tuning, motivating us to freeze these layers. At the rank level, although trailing components in deeper layers generally undergo larger changes, leading components also exhibit non-negligible changes and dominate the overall parameter structure. We therefore concentrate adaptation on the leading spectral components of deeper layers, allowing us to modify the dominant parameter structure while preserving the highly stable shallow layers.
Q2. Distinction from LoRA-XS
Although our method shares a similar parameterization and update space with LoRA-XS, their motivations and adaptation strategies differ. LoRA-XS provides a general SVD-based PEFT approach that adapts all layers, whereas our method is motivated by the observed layer-wise spectral stability and selectively adapts deeper layers as a functional validation of our findings.
Q3. Generalization Beyond Encoder-Based Models
We conducted additional experiments using Qwen2.5-1.5B, a modern decoder-only LLM, on Countdown and Knights and Knaves reasoning tasks. Exchanging the leading spectral components between pretrained and fine-tuned models causes no noticeable performance degradation. These results provide additional evidence for the preservation and interchangeability of pretrained spectral structures beyond encoder-based models and GLUE tasks.
I thank the authors for their clarifications and additional experiments, which have addressed some of my concerns. However, I believe two key issues remain insufficiently addressed.
First, the novelty remains limited. Although the authors clarify that their primary contribution lies in the systematic investigation of spectral stability and highlight the different stability patterns across shallow and deep layers, the proposed adaptation strategy still represents an incremental extension of existing SVD-based PEFT methods. The layer-aware selection strategy does not substantially distinguish the proposed method from existing approaches.
Second, the motivation for the proposed adaptation strategy remains unclear. If the stability of shallow layers motivates freezing their parameters, why are the relatively stable leading spectral components in deeper layers prioritized for adaptation, rather than the trailing components that exhibit larger changes during fine-tuning? The observation that leading components dominate the pretrained parameter structure does not necessarily establish their importance for downstream adaptation. The current explanation does not sufficiently justify this distinction or establish a convincing connection between the observed spectral stability and the proposed adaptation strategy.
W1. Misplaced Expectation of Algorithmic Novelty
We respectfully believe that the reviewer places disproportionate emphasis on a supplementary component while overlooking the central contributions of our work.
Our paper focuses on understanding learning systems, rather than proposing a novel PEFT algorithm. Our main contribution is the systematic discovery and characterization of stable spectral subspaces in pretrained models, supported by extensive experiments across architectures, modalities, tasks, and pretraining scales.
The PEFT experiments serve only as supplementary functional validation, not as a core contribution. We would even be willing to remove the entire PEFT component, as this would not affect our main findings or conclusions. Therefore, evaluating the novelty of our work primarily through comparisons with existing PEFT methods misses the central scientific contribution of this paper.
W2. Leading vs. Trailing Subspaces for Adaptation
We do not claim that leading spectral subspaces are inherently better adaptation directions than trailing ones. To clarify this point, we added an analysis in Section O of the Appendix (page 30). The results show that the choice of spectral subspace has a relatively limited impact on performance, whereas layer-wise allocation plays a more important role.
We will also revise the relevant statements to avoid implying that leading subspaces are inherently superior for fine-tuning. Our finding concerns the layer-wise structure of spectral stability, not the superiority of a particular adaptation subspace.
Response (Part 1)
Weaknesses
We agree that the original manuscript did not sufficiently discuss related SVD-based PEFT methods. We have therefore expanded the discussion of LoRA-XS, PiSSA, MiLoRA, and SVFT. These methods all exploit the spectral structure of pretrained weights but differ in their adaptation strategies: LoRA-XS freezes SVD-derived singular vectors and learns an intermediate coefficient matrix; PiSSA initializes trainable LoRA components using principal singular components; MiLoRA focuses on the trailing spectral components; and SVFT constrains updates through different spectral-space parameterization patterns.
We acknowledge that our spectral-constrained adaptation shares a similar parameterization with LoRA-XS. Our main contribution, however, is not a fundamentally new spectral adaptation parameterization, but a systematic investigation of spectral stability in pretrained models. We show that spectral stability varies substantially across network depth, with shallow layers exhibiting greater stability than deeper layers. Based on this finding, we adapt only selected spectral components in deeper layers while keeping shallow layers frozen, whereas LoRA-XS applies its adaptation across all layers. Our experiments demonstrate that this layer-aware strategy achieves effective downstream adaptation, providing further evidence for the functional significance of the observed layer-wise spectral stability.
We have revised the manuscript to better position our work relative to these methods and clarify that its primary contribution is understanding the stability, transferability, and functional implications of pretrained spectral structures.
Our adaptation strategy is motivated by a hierarchical analysis of spectral stability at both the layer and rank levels.
At the layer level, we find that pretrained parameters in shallow layers remain highly stable during fine-tuning, exhibiting relatively small changes across their spectral components. This motivates us to freeze shallow layers and preserve their pretrained parameter structures.
At the rank level, we find that deeper layers undergo more substantial parameter changes, with trailing spectral components generally exhibiting greater changes than leading components. Nevertheless, leading spectral components also undergo non-negligible changes during fine-tuning. Importantly, leading components account for the dominant part of the pretrained parameter structure. Therefore, we choose to adapt these components in deeper layers, as updating the dominant spectral components provides an effective way to modify the principal parameter structure while maintaining a structured and parameter-efficient adaptation space.
Together, these findings motivate a selective adaptation strategy that preserves stable pretrained parameters while updating only the spectral components that undergo substantial changes during fine-tuning. Rather than introducing a new adaptation method, our goal is to investigate whether selectively updating these components, while keeping the more stable parameters fixed, can achieve comparable downstream performance to conventional fine-tuning.
To further examine whether our observations extend beyond encoder-based architectures, we conducted additional experiments using Qwen2.5-1.5B, a modern decoder-only language model, on two reasoning tasks: Countdown (1,024 problems) and Knights and Knaves (500 problems).
Specifically, we exchanged the top-K spectral components between pretrained and fine-tuned models and evaluated the resulting task performance. As shown in the new results, exchanging the leading spectral components (K = 1, 5, 10, 20, and 30) leads to no noticeable performance degradation on either task.
These experiments provide additional evidence that the observed spectral stability and component-exchange robustness extend beyond encoder-based language models to a modern decoder-only LLM and two reasoning tasks. We acknowledge that these results do not establish the effectiveness of our proposed adaptation method across decoder-only architectures; rather, they provide complementary evidence for the broader applicability of the spectral phenomena investigated in this work.