acceptodds
Withdrawn from ICLR 2027

Pretraining Induces Stable Spectral Subspaces for Downstream Task Adaptation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

87 views

Abstract

Finetuning pretrained models often exhibits a low-dimensional structure in parameter space, with prior work focusing on where adaptation occurs. We instead investigate which parameter subspaces remain stable across finetuning and downstream tasks. We analyze pretrained weight matrices across vision and language models and find that leading spectral subspaces remain largely unchanged during finetuning, with stability consistently stronger in earlier layers. This stability emerges progressively during pretraining and increases with pretraining dataset size, suggesting that downstream adaptation largely preserves the leading spectral structure formed during pretraining. We further show that downstream adaptation can be restricted to pretrained leading spectral subspaces in deeper layers, achieving performance close to full finetuning while updating only 0.2% of model parameters on GLUE. Our findings offer a new parameter-space perspective on how pretraining shapes downstream adaptation, highlighting leading spectral subspaces as stable structure inherited across tasks.

Primary area:
optimization
Keywords:
Pre-trained Models, Subspace, Fine tuning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_mDjLOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates which parameter subspaces remain stable during finetuning. Through experiments across vision and language models, the authors find that leading spectral subspaces are largely preserved, particularly in shallower layers, and that their stability increases with pretraining data scale. Based on these observations, the authors show that downstream adaptation can be restricted to the leading spectral subspaces of deeper layers while maintaining competitive performance. The authors claim that these findings provide a parameter-space perspective on how pretraining shapes downstream adaptation.

Critical strengths

1.Understanding the spectral relationship between pretrained and finetuned models remains an open problem, and the paper offers an interesting perspective by investigating stable rather than changing parameter subspaces.

2.The empirical analysis is relatively comprehensive. The authors investigate spectral stability across different architectures, layers, and tasks, supported by cross-task singular-vector swapping, pretraining-scale experiments, and analyses of spectral rank and layer selection. These experiments provide useful empirical insights.

Critical weaknesses

1.The proposed adaptation closely resembles LoRA-XS [1], which also freezes SVD-derived projections and trains only an intermediate matrix. However, LoRA-XS is neither cited nor discussed. The paper also insufficiently positions its findings relative to PiSSA, SVFT, and MiLoRA [2], raising concerns about the novelty of its methodological insights.

2.Section 3.2 does not establish a convincing link between the observed spectral stability and the proposed adaptation method. The authors find that leading singular subspaces remain stable during finetuning, but they subsequently restrict adaptation to these same subspaces. Stability alone does not establish that these are the most effective update directions. The proposed method therefore lacks sufficient justification from the preceding analysis.

3.Although the paper examines multiple vision and language architectures, its key functional and cross-task evaluations focus on encoder-based language models and GLUE tasks, while controlled pretraining experiments use a small-scale vision model. The absence of modern decoder-only LLMs and more challenging downstream tasks limits the generalizability of the conclusions.

[1] Bałazy et al. LoRA-XS: Low-Rank Adaptation with Extremely Small Number of Parameters. ArXiv 2024.

[2] Wang et al. MiLoRA: Harnessing Minor Singular Components for Parameter-Efficient LLM Finetuning. NAACL 2025.

Questions for authors

1.How is the proposed adaptation method motivated by the observed spectral stability? Why should adaptation target stable leading subspaces rather than directions that change more substantially?

2.How does the proposed method differ from LoRA-XS? Given their effectively equivalent update spaces, what distinct methodological insight does the proposed parameterization provide, and why is LoRA-XS not discussed?

3.How well do the findings generalize to modern LLMs and more challenging tasks? Additional evidence from decoder-only LLMs and more diverse downstream adaptation settings would strengthen the paper's broader claims.

4 comments
AuthorsOpenReview ↗

Response (Part 1)

Weaknesses

W1. Positioning Against SVD-Based PEFT Methods

We agree that the original manuscript did not sufficiently discuss related SVD-based PEFT methods. We have therefore expanded the discussion of LoRA-XS, PiSSA, MiLoRA, and SVFT. These methods all exploit the spectral structure of pretrained weights but differ in their adaptation strategies: LoRA-XS freezes SVD-derived singular vectors and learns an intermediate coefficient matrix; PiSSA initializes trainable LoRA components using principal singular components; MiLoRA focuses on the trailing spectral components; and SVFT constrains updates through different spectral-space parameterization patterns.

We acknowledge that our spectral-constrained adaptation shares a similar parameterization with LoRA-XS. Our main contribution, however, is not a fundamentally new spectral adaptation parameterization, but a systematic investigation of spectral stability in pretrained models. We show that spectral stability varies substantially across network depth, with shallow layers exhibiting greater stability than deeper layers. Based on this finding, we adapt only selected spectral components in deeper layers while keeping shallow layers frozen, whereas LoRA-XS applies its adaptation across all layers. Our experiments demonstrate that this layer-aware strategy achieves effective downstream adaptation, providing further evidence for the functional significance of the observed layer-wise spectral stability.

We have revised the manuscript to better position our work relative to these methods and clarify that its primary contribution is understanding the stability, transferability, and functional implications of pretrained spectral structures.

W2. Motivation for Spectral Adaptation

Our adaptation strategy is motivated by a hierarchical analysis of spectral stability at both the layer and rank levels.

At the layer level, we find that pretrained parameters in shallow layers remain highly stable during fine-tuning, exhibiting relatively small changes across their spectral components. This motivates us to freeze shallow layers and preserve their pretrained parameter structures.

At the rank level, we find that deeper layers undergo more substantial parameter changes, with trailing spectral components generally exhibiting greater changes than leading components. Nevertheless, leading spectral components also undergo non-negligible changes during fine-tuning. Importantly, leading components account for the dominant part of the pretrained parameter structure. Therefore, we choose to adapt these components in deeper layers, as updating the dominant spectral components provides an effective way to modify the principal parameter structure while maintaining a structured and parameter-efficient adaptation space.

Together, these findings motivate a selective adaptation strategy that preserves stable pretrained parameters while updating only the spectral components that undergo substantial changes during fine-tuning. Rather than introducing a new adaptation method, our goal is to investigate whether selectively updating these components, while keeping the more stable parameters fixed, can achieve comparable downstream performance to conventional fine-tuning.

W3. Generalization to Decoder-Only LLMs

To further examine whether our observations extend beyond encoder-based architectures, we conducted additional experiments using Qwen2.5-1.5B, a modern decoder-only language model, on two reasoning tasks: Countdown (1,024 problems) and Knights and Knaves (500 problems).

Specifically, we exchanged the top-K spectral components between pretrained and fine-tuned models and evaluated the resulting task performance. As shown in the new results, exchanging the leading spectral components (K = 1, 5, 10, 20, and 30) leads to no noticeable performance degradation on either task.

TaskBefore ExchangeTop-1Top-5Top-10Top-20Top-30
Countdown (1,024 problems)71.48%71.88%72.07%72.46%73.05%72.07%
Knights and Knaves (500 problems)23.00%23.20%23.20%23.20%23.00%23.00%

These experiments provide additional evidence that the observed spectral stability and component-exchange robustness extend beyond encoder-based language models to a modern decoder-only LLM and two reasoning tasks. We acknowledge that these results do not establish the effectiveness of our proposed adaptation method across decoder-only architectures; rather, they provide complementary evidence for the broader applicability of the spectral phenomena investigated in this work.

AuthorsOpenReview ↗

Response (Part 2)

Questions

Q1. Justification for Leading-Subspace Adaptation

Our adaptation strategy is motivated by hierarchical analyses of spectral stability at both the layer and rank levels. At the layer level, shallow layers exhibit small changes across their spectral components during fine-tuning, motivating us to freeze these layers. At the rank level, although trailing components in deeper layers generally undergo larger changes, leading components also exhibit non-negligible changes and dominate the overall parameter structure. We therefore concentrate adaptation on the leading spectral components of deeper layers, allowing us to modify the dominant parameter structure while preserving the highly stable shallow layers.

Q2. Distinction from LoRA-XS

Although our method shares a similar parameterization and update space with LoRA-XS, their motivations and adaptation strategies differ. LoRA-XS provides a general SVD-based PEFT approach that adapts all layers, whereas our method is motivated by the observed layer-wise spectral stability and selectively adapts deeper layers as a functional validation of our findings.

Q3. Generalization Beyond Encoder-Based Models

We conducted additional experiments using Qwen2.5-1.5B, a modern decoder-only LLM, on Countdown and Knights and Knaves reasoning tasks. Exchanging the leading spectral components between pretrained and fine-tuned models causes no noticeable performance degradation. These results provide additional evidence for the preservation and interchangeability of pretrained spectral structures beyond encoder-based models and GLUE tasks.

I thank the authors for their clarifications and additional experiments, which have addressed some of my concerns. However, I believe two key issues remain insufficiently addressed.

First, the novelty remains limited. Although the authors clarify that their primary contribution lies in the systematic investigation of spectral stability and highlight the different stability patterns across shallow and deep layers, the proposed adaptation strategy still represents an incremental extension of existing SVD-based PEFT methods. The layer-aware selection strategy does not substantially distinguish the proposed method from existing approaches.

Second, the motivation for the proposed adaptation strategy remains unclear. If the stability of shallow layers motivates freezing their parameters, why are the relatively stable leading spectral components in deeper layers prioritized for adaptation, rather than the trailing components that exhibit larger changes during fine-tuning? The observation that leading components dominate the pretrained parameter structure does not necessarily establish their importance for downstream adaptation. The current explanation does not sufficiently justify this distinction or establish a convincing connection between the observed spectral stability and the proposed adaptation strategy.

W1. Misplaced Expectation of Algorithmic Novelty

We respectfully believe that the reviewer places disproportionate emphasis on a supplementary component while overlooking the central contributions of our work.

Our paper focuses on understanding learning systems, rather than proposing a novel PEFT algorithm. Our main contribution is the systematic discovery and characterization of stable spectral subspaces in pretrained models, supported by extensive experiments across architectures, modalities, tasks, and pretraining scales.

The PEFT experiments serve only as supplementary functional validation, not as a core contribution. We would even be willing to remove the entire PEFT component, as this would not affect our main findings or conclusions. Therefore, evaluating the novelty of our work primarily through comparisons with existing PEFT methods misses the central scientific contribution of this paper.

W2. Leading vs. Trailing Subspaces for Adaptation

We do not claim that leading spectral subspaces are inherently better adaptation directions than trailing ones. To clarify this point, we added an analysis in Section O of the Appendix (page 30). The results show that the choice of spectral subspace has a relatively limited impact on performance, whereas layer-wise allocation plays a more important role.

We will also revise the relevant statements to avoid implying that leading subspaces are inherently superior for fine-tuning. Our finding concerns the layer-wise structure of spectral stability, not the superiority of a particular adaptation subspace.

Reviewer_oW7vOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the stability of parameter subspaces in pretrained models during fine-tuning. Through SVD analysis across vision and language models, the authors find that leading spectral subspaces remain largely unchanged, particularly in earlier layers, and are shared across downstream tasks. They further show that larger pretraining datasets produce more stable spectral structures. Based on these observations, they propose a spectrally restricted fine-tuning approach that achieves performance close to full fine-tuning while updating fewer parameters.

Critical strengths
  1. The paper investigates spectral stability across different vision and language models and network layers. It studies how spectral structures remain stable during fine-tuning.

  2. The experiments provide some insights into how pretrained spectral structures are preserved across tasks and how they can support parameter-efficient downstream adaptation.

Critical weaknesses

A major concern is that the observed spectral stability may simply reflect small parameter changes during training and fine-tuning, rather than a distinctive property of pretrained spectral structures:

  1. In Fig.2, the high spectral similarity between pretrained and fine-tuned models may result from relatively small weight updates during fine-tuning, i.e. ∣∣Wpt∣∣≈∣∣Wft∣∣||W^{pt}||\approx ||W^{ft}||

  2. In Fig.4, the increasing spectral stability during pretraining may simply reflect the natural convergence of model weights, rather than the emergence of meaningful or transferable spectral structures.

  3. In Fig.3, due to the small weight difference between pretrained and fine-tuned models, interchanging their spectral components won't cause much difference. But replacing them with 0 or random values obviously will compromise the performance.

Questions for authors
  1. Could you check the relative change of weights of the fine-tuned model to the pretrained model before drawing conclusions about spectral stability?

  2. Tab.1 doesn't provide clear evidence of why the proposed method works. Could you check LoRA with a lower rank and compare the performance with it while maintaining the same number of parameters?

2 comments

Weaknesses

W1. Spectral Stability vs. Parameter Changes

We have added further analyses in Appendix Section M (page 28) to examine whether the observed spectral similarity between pretrained and fine-tuned models can be explained solely by the magnitude of parameter updates.

We first quantified the relative parameter changes by computing the differences between pretrained and fine-tuned weight matrices and comparing the singular values of the update matrices with those of the original pretrained weights. As expected, smaller relative parameter changes generally correspond to higher similarity between the leading singular subspaces. However, we also observed cases in which substantial parameter changes coexist with high spectral similarity, indicating that update magnitude alone is insufficient to explain the preservation of pretrained spectral structures.

We further examined the geometric relationship between parameter updates and pretrained spectral directions. Interestingly, even when parameter updates are substantial, their dominant singular directions can remain highly aligned with those of the original pretrained weights. This suggests that the preservation of spectral structures depends not only on the magnitude of parameter updates but also on their geometric alignment with the pretrained spectral structure. Therefore, the observed spectral stability cannot be attributed solely to small parameter changes.

W2. Spectral Stability vs. Weight Convergence

We agree that the increasing spectral similarity during pretraining, as shown in Fig. 4A and 4D, may partly reflect the natural convergence of model parameters. However, parameter convergence alone does not fully explain the differences in spectral stability observed across pretraining data scales during subsequent training.

Specifically, Fig. 4B, 4C, 4E, and 4F show that models pretrained on larger datasets exhibit greater preservation of their spectral structures under subsequent training on new tasks or data. Importantly, our additional experiments with matched optimization steps (Appendix Section N, page 30) demonstrate that this difference persists even when the number of pretraining updates is controlled.

Together, these results suggest that pretraining data scale contributes to the formation of transferable spectral structures beyond its effect on optimization duration. Thus, our findings concern not only the convergence of model parameters but also the robustness of the resulting spectral structures to subsequent training.

W3. Interpretation of Component-Exchange Robustness

We acknowledge that the small parameter differences between pretrained and fine-tuned models may contribute to the limited performance changes observed when exchanging their spectral components. However, regardless of whether this robustness arises from the similarity between pretrained and fine-tuned parameters or from the preservation of specific spectral structures, the key implication remains the same: the pretrained parameter subspaces already capture information relevant to downstream tasks and can be directly transferred to support downstream performance with only limited adaptation. In contrast, replacing these components with zeros or random values disrupts the pretrained structures and leads to performance degradation. Together, these results demonstrate the transferability of pretrained parameter subspaces to downstream tasks, which is the central finding we aim to establish in Fig. 3.

Questions

Q1. Relative Parameter Changes

We have conducted additional analyses of relative weight changes between pretrained and fine-tuned models (Appendix Section M, page 28). Although smaller relative changes generally correspond to higher spectral similarity, we also observe substantial parameter changes with preserved spectral structures, indicating that weight-change magnitude alone cannot fully explain spectral stability.

Q2. Comparison with Parameter-Matched LoRA

We have added lower-rank LoRA baselines, with the number of trainable parameters matched closely to that of our method.

MethodRTEMRPCSTS-BCoLASST-2QNLIMNLIQQPAverage
Our Method (0.36M)84.1288.4888.8568.1395.5392.0687.4589.2286.73
LoRA (r=6, 0.33M)85.9289.9591.4569.9595.6494.0390.3191.8288.63

The results show that lower-rank LoRA achieves comparable performance, suggesting that the performance of our method is largely comparable under a similar parameter budget.

Thank you for the additional experiments. The new analyses partially address my concerns. Here are my concerns:

  • Appendix M reports a negative correlation between relative parameter changes and spectral similarity, which supports my original concern that small weight updates may explain much of the observed stability.
  • Appendix N does not directly address whether the increasing stability in Fig. 4A/D results from weight convergence.
  • My concern regarding Fig. 3 remains unresolved.
  • LoRA consistently outperforms the proposed method using fewer trainable parameters.

I tend to keep my score.

Reviewer_12mMOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This submission explores what fine-tuning preserves in its weights with subspace similarity measure suite. To be specific, by using the SVD and principal-angle measurements this submission figure out that 1) pre-training establishes good function between input and output with weights and 2) fine-tuning preserves the leading weight subspace in earlier layers across diverse tasks, architectures, and modalities. Based on above, the effective adaptation can be achieved by fine-tune the spectral components at deeper layers. These empirical results extend the understanding of the SVFT methods.

Critical strengths
  • New empirical observations: This submission provides useful empirical and general observation for the adaptation community. To be specific, it examines both vision and language model and almost all the model layers to strengthen the findings.
  • The analysis are comprehensive and sound: The introducing of the principal-angle analysis are interesting for the deep learning community. Beyond the similarity analysis, pretrained-vector restoration, cross-task swapping, and masking provide stronger proof than similarity heatmaps.
  • Interesting failure case to reveal the limitations`: random label fine-tuning breaks the observation on weight subspace which means such phenomenon depends on the data and fine-tuning settings.
Critical weaknesses
  • The pre-training diagnose does not support larger dataset promote the stable spectral subspace. These experiments are conducted with fixed epoch which indicates that larger datasets entail more optimization steps.
  • Incomplete mechanistic explanation: This submission does not explain why this structure stays stable.
Questions for authors
  • Improvement significant: Does the GLUE results hold across multiple runs?
  • What if the adaptation is performed within leading spectral subspaces compare with trailing and random subspaces?
2 comments

Weaknesses

W1. Data Scale vs. Training Duration

To disentangle the effects of pretraining data scale and optimization duration, we conducted additional experiments (Section N in Appendix, page 30) by matching the number of optimization steps across different pretraining dataset sizes, followed by finetuning on the same downstream tasks. The results show that extending the training of models pretrained on smaller datasets indeed improves their spectral stability, as reflected by higher similarity after finetuning. However, even with matched optimization steps, models pretrained on larger datasets consistently exhibit higher similarity. These findings indicate that although longer pretraining contributes to spectral stability, the benefits of larger pretraining datasets cannot be explained solely by increased optimization steps. Instead, larger data scale provides an additional contribution to the formation of stable spectral subspaces.

W2. Mechanism of Spectral Stability

We acknowledge that the current study does not establish a complete and rigorous mechanistic explanation for spectral stability. Instead, our analysis provides an exploratory interpretability perspective on this phenomenon, and we leave a more rigorous mechanistic investigation for future work.

Questions

Q1. Reproducibility Across Random Seeds

The reported GLUE results are averaged over three independent random seeds. The corresponding standard deviations are reported below to demonstrate the variability and consistency of the results across runs.

MethodRTEMRPCSTS-BCoLASST-2QNLIMNLIQQPAverage
Our Method84.12 ± 0.0188.48 ± 0.2588.85 ± 0.0568.13 ± 0.9995.53 ± 0.1792.06 ± 0.0287.45 ± 0.0489.22 ± 0.1286.73 ± 0.15

Q2. Comparison Across Spectral Subspaces

We have added experiments (Section O in Appendix, page 30) comparing adaptation within different spectral subspaces. The results exhibit some fluctuations across subspaces, but the overall performance differences are relatively modest. In comparison, the choice of adaptation layer depth has a more substantial impact on downstream performance.

Regarding random-subspace adaptation, our current parameterization does not directly support the selection of arbitrary subspaces, as it optimizes an intermediate parameter block while keeping the spectral bases fixed.

Thanks for your detailed response. And the response address my concerns on optimization steps. However, according to the response to other reviews, I partially agree with oW7v. The subspace structure preservation could not be a common interesting phenomenon. It may result from small weight changes. The authors can resort to out-of-domain data for further exploration. Thus the claim of this submission should be revisited. I will decrease my intial assesement.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.