acceptodds
Under review as a conference paper at ICLR 2027

Are Parameter-efficient Fine-tuning Methods Really Different?

Abstract

Parameter-efficient fine-tuning (PEFT) offers many parameterizations, yet their methodological and functional differences remain unclear. We compare six methods in language and diffusion models to examine how their parameterizations relate to task performance, forgetting, and changes in pretrained weight geometry. Motivated by the spectrum-preserving design of orthogonal fine-tuning (OFT), we first ask whether spectral preservation is itself important for adaptation and retention. We find that the selected LoRA-family methods also approximately preserve pretrained geometry, and that restoring their slightly drifted singular-value spectra largely preserves task performance, questioning the necessity of explicit geometric preservation. Beyond this, we observe that some methods exhibit distinct adaptation–retention trade-offs that vary across settings: LoRA most consistently limits forgetting at competitive performance, DoRA achieves higher mean task scores than LoRA in most comparisons, while PiSSA often incurs greater retention costs. Further intervention experiments suggest that while performance gains from different PEFT methods can be attributed to modifications in different groups of spectral components, we consistently find that restoring dominant rather than intermediate or trailing components produces the largest mean reduction in general-text NLL or base-image drift. Together, these results motivate evaluating geometric constraints through their functional consequences rather than preservation alone.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Parameter Efficient Fine-Tuning, Large Language Models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_adDBOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper makes a comparative and diagnostic contribution rather than introducing a new method or establishing a new performance record. It investigates how different parameter-efficient fine-tuning (PEFT) methods compare, examining six methods across language and diffusion models in terms of task performance, forgetting, and changes in pre-trained weight geometry.

The findings suggest that different PEFT methods can achieve similar task performance while exhibiting substantially different levels of forgetting. LoRA often preserves pre-trained behavior most effectively, DoRA frequently achieves somewhat higher task scores, and PiSSA tends to incur greater retention costs. The paper also finds that explicitly preserving pre-trained weight geometry does not consistently lead to better outcomes.

Critical strengths

The paper provides an effective empirical comparison of different PEFT methods, with several interesting observations regarding the relationship between adaptation performance, forgetting, and pre-trained weight geometry.

In particular, the authors show that restoring pre-trained singular values often preserves adaptation performance. They also employ selective spectral interventions to investigate which components of the weight matrices contribute to adaptation and forgetting. These experiments provide interesting insights into the behavior of different PEFT methods.

Critical weaknesses

My main reservation is the lack of a significant theoretical contribution. The paper is primarily comparative and diagnostic, without introducing a new method or establishing a new performance record.

I would support acceptance if this type of empirical and comparative contribution is considered sufficient under ICLR's standards. The experimental observations are interesting, but the overall significance of the contribution should be evaluated in that context.

Questions for authors

None.

3 comments

Empirical contribution. We appreciate the reviewer’s positive assessment and agree that our main contribution is empirical. ICLR’s reviewer guide recognizes new empirical knowledge as a contribution.

Motivation. Many PEFT variants claim gains over LoRA, but these gains do not always stand the test of time. We ask which designs improve performance or reduce forgetting under equal tuning budgets, and which patterns of adaptation and forgetting hold across methods.

Our findings.

  1. Tuned LoRA most frequently limited forgetting at competitive task scores (Table 14). In the AUC sense, DoRA achieves the best performance-forgetting tradeoff rank, followed by LoRA (Figures 3c,d and 4b). OFT and HRA preserved weight geometry more closely than LoRA, yet neither showed a consistent advantage in task performance or retention (§§3–4).
  2. There is no consistent conclusion about which spectral components matter most for task performance. However, the top spectral components consistently matter most for forgetting (§5).

Thanks for your comments. I therefore maintain my scores.

Dear Reviewer adDB,

Thank you for taking the time to review our work and consider our response.

Kind regards, The Authors

Reviewer_AAwMOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper compares multiple PEFT methods, including LoRA, DoRA, PiSSA, MiLoRA, OFT and HRA, studies their differences in target task performance, pretrained knowledge preservation and changes in weight geometry. The experiments cover domain adaptation tasks on math and coding with two language models, as well as diffusion personalization. The experiments use method specific LR choices, nearly matched parameter budgets and multiple random seeds for training.

Besides adaptation-retention trade-offs, the paper also analyzes singular-value displacement and hyper-spherical energy preservation. Two other spectral intervention experiments are conducted: (1) restoring pretrained singular values while retaining adapted singular bases, and (2) selectively restoring top/middle/tail spectral components.

The main conclusions of the paper include: (1) singular-value restoration largely preserves task performance; (2) different PEFT methods can achieve comparable downstream performance while exhibiting different retention behavior; (3) and restoring dominant spectral components consistently mitigate forgetting (in general-text NLL or base-image drift). Based on the observations above, the authors question the necessity of explicit geometric preservation, and argue for evaluating geometric constraints through their functional consequences.

Critical strengths
  1. The comparison is systematic and well controlled. Six representative PEFTs are selected for comparison, controlled with similar trainable parameter budget. A fair method-specific learning rate tuning is performed, and multiple training seeds are used for repeated experiments. These robustness checks make the reported adaptation-retention performance pattern more convincing. These experiment provide solid empirical evidence, that different PEFT methods can achieve comparable downstream performance while exhibiting different retention behavior, which is consistent with the observation of some of the previous research.

  2. Interesting spectral intervention for retention with consistency checks. Besides comparing the performance, and connecting it with geometric analysis, the research directly perform intervention on the singular values and different groups of spectral component, trying to mitigate forgetting. This experiment comes up with an interesting observation, that dominant-component restoration achieve consistently retention improvement across methods, capacities, and seeds. For some of the method, this dominant-component restoration only cause negligible target task performance drop.

Critical weaknesses
  1. Concerns about the interpretation of spectral interventions. (Section 3.) The singular value restoration pairs pretrained and adapted spectral vectors by sorted indices, such vector pairs are not necessarily the same pretrained singular directions. Similarly, the component restoration experiments mix pretrained and adapted singular bases, potentially introducing additional modifications rather than simply reverting the original updates. This could be easily happened due to the sensitivity of SVD to numerical precision. Therefore, although the observed mitigation in NLL or base-image drift are interesting, they do not necessarily establish how exactly forgetting is caused. The claim of approximate HE preservation may be overstated. For example, POET[1] (Figure 7) reports HE varies by roughly 0.02–0.08%, which is comparable to the changes reported here (up to 0.0098%). A small relative HE change does not necessarily imply negligible geometric or functional effects.

  2. The relationship to closely related work is insufficiently discussed. Prior studies have already investigated singular-value/vector restoration and spectral interventions in relation to forgetting (e.g., He & Cao; Jin et al.; Shuttleworth et al.), but the paper does not clearly discuss the difference between its finding and theirs. The paper mentioned a similar study, PEFT-Arena (Huang et al.), which also studies adaptation-retention tradeoff of PEFTs. However, the paper only mentioned its shared learning rates and single-seed evaluations, without discussing the experiment setting, metrics, and geometric analyses. A more in-depth discussion of these related research is important to clarify the contribution of the paper.

  3. Concerns about limited evidence of meaningful adaptation gains. The math experiment does not report base performance of GSM8K for both backbones, making the actual adaptation gains unclear. The GSM8K score of Qwen2.5-7B-base reported by the Qwen official report[2] is 85.4%, which is actually higher than the adapted score in this paper, making the actual adaptation gains unclear. In coding, Qwen2.5-7B shows only marginal improvements, with Llama-3.1-8B showing more substantial gains. This raises concerns about whether the evaluation settings sufficiently support the conclusions on adaptation-retention trade-offs, as well as the task performance retention with spectral intervention. Also, I wonder if the authors have considered evaluating one of the instruction-tuned models (e.g., Llama-3.1-8B-Instruct or Qwen2.5-7B-Instruct).

[1] Reparameterized LLM Training via Orthogonal Equivalence Transformation, Qiu et al., 2025

[2] Qwen2.5 Technical Report, 2024

Questions for authors

Q1. How do the authors justify the interpretation of spectral restoration as reversing the original updates, given that it may introduce additional changes to pretrained singular directions? Analysis of the directional similarity of paired singular vectors could possible help us better understand what the spectral restoration is actually doing. Also, is it possible to assess the functional significance of the small HE changes?

Q2. Could the authors clarify how the findings differ from prior related works mentioned in Weakness 2, particularly in terms of experimental settings, evaluation metrics, and conclusions?

Q3. Could the authors report the unadapted GSM8K performance of both backbones under the same evaluation protocol? How substantial are the actual adaptation gains in the math experiments?

Q4. The language model experiments mainly uses FineWiki-200 NLL to measure forgetting. Have the authors evaluated retention on any benchmarks that are related to the general knowledge or capability of LLMs? It is unclear whether the observed NLL changes is consistent with the forgetting of pretrained capabilities.

8 comments

Thank you for the thoughtful questions about how we interpret spectral restoration and measure forgetting. We have conducted these experiments and polished our results. The major revision are highlighted in red in the updated version.

W1 + Q1

1.1 Singular-value orders can change after fine-tuning

Yes, ranking can be changed but doesn't affect the results and conclusions. This is because, the restoration operation keeps tuned directions fixed and gives the closest matrix with the pretrained spectrum (Proposition E.1).

1.2 How do mixing and numerical precision affect group restoration?

Group-level comparisons are less sensitive to changes in the ranks of individual directions. We observed that the mean principal angle between pretrained and tuned right singular subspaces was below 1° for each of the three groups, suggesting that the groups remained closely aligned even if individual directions changed rank.

In all tested checkpoints, restoring the dominant group reduced forgetting more than reconstruction and casting alone, making numerical error an unlikely explanation for the observed reduction (Appendix C.1).

1.3 Do small HE changes matter for performance?

HE changes has no strong correlation to the performance. OFT and HRA preserved HE more closely than LoRA without a consistent advantage on either measure.


W2 + Q2 — Comparison with previous work on settings, metrics, and findings

We compare six PEFT methods on language and image tasks, measuring task scores alongside FineWiki ΔNLL or base-image drift (§2, Appendix A.3). Section 1.2 and Appendix F give details.

Study and aimExperimental settingEvaluation metricsConclusions
He & Cao [2] analyze full finetuning weight change using SVDFully tuned LLMs, rescaled spectrum restorationAccuracy, singular-vector rotationRestoring the rescaled spectrum largely preserved accuracy. Leading directions rotated little, suggesting adaptation mainly involved other directions.
Jin et al. [3] — explain RL and SFT differenceFull-parameter SFT followed by RL, progressive restoration of leading directionsAccuracyRL recovered some accuracy under changed rules. Full-parameter SFT gains were concentrated mainly in leading directions.
Shuttleworth et al. [4] — explain LoRA forgetting compared with Full FTLoRA versus full tuning, attenuation of left singular directions with no close pretrained match (“intruders”)Accuracy, pretraining pseudo-lossAttenuating LoRA intruders reduced forgetting with little loss in task accuracy.
PEFT-Arena [5] — compare adaptation and retention across PEFT methodsCOT style Math and medical SFT, one shared LR and one seedTask scores, IFEval, NQ, BBH, weight and activation geometryOFT showed a favorable adaptation–retention trade-off, which the authors linked to preservation of weight and activation geometry.
Our work — identify what holds across PEFT methodsSix methods on language and diffusion tasks, per-method LR tuning over three seeds, spectral group restorationTask scores, FineWiki ΔNLL, image drift(1) There is no consistent conclusion for task performance. However, the top spectral components consistently matters the most on forgetting. (2) Tuned LoRA forgot least in 9/12 settings at competitive task scores. (3) OFT and HRA do not show consistently better accuracy-forgetting trade-off

Why their conclusions cannot explain our results:

  1. He & Cao [2] and Jin et al. [3]. He & Cao observed little rotation in leading directions and suggested adaptation mainly involved other components. Jin et al. traced full-parameter SFT gains mainly to leading directions. In our PEFT comparison, however, there is no consistent conclusion for task performance, while the top spectral components consistently matters the most on forgetting.
  2. Shuttleworth et al. [4]. The exact input-side rotations used by OFT and HRA introduce no intruder directions, yet both methods often forgot more than LoRA in our experiments. Intruders alone therefore cannot explain the differences in forgetting across these PEFT methods.
  3. PEFT-Arena [5]. Their shared-LR comparison favored OFT. In our setup, OFT showed no consistent trade-off advantage over LoRA.

W3 + Q3

3.1 Base model GSM8K scores and Qwen official score

  1. Gains over base. Base Qwen scored 4.40% and base Llama 0.00% on zero-shot GSM8K. Fine-tuning improves more than 60 percentage points accuracy for all methods (Table 2a).
  2. Qwen’s official score. Qwen’s reported 85.4% uses 4-shot prompting [6]. Our evaluation was zero-shot.

3.2 Did tuning also improve coding?

Llama gained over 3 percentage points on both coding benchmarks, while Qwen changed little. Base Qwen already scored 13–20 points above base Llama, which may help explain its smaller gains (Table 2b).

3.3 Will you test an instruction-tuned backbone?

Thanks for the suggestion. We’re testing an instruction-tuned backbone and will update our results once finished.


Q4 — Does benchmark retention agree with FineWiki NLL?

Yes, both measures favored LoRA over OFT on average. LoRA achieves higher on the PEFT-Arena general score [5] and had lower FineWiki-200 ΔNLL. The disjoint FineWiki-10,000 evaluation also favored LoRA at medium capacity (Appendix B.1, Tables 6 and 9).


References

  • [1] Reparameterized LLM Training via Orthogonal Equivalence Transformation
  • [2] Understanding Post-Training Structural Changes in Large Language Models
  • [3] RL Fine-Tuning Heals OOD Forgetting in SFT
  • [4] LoRA vs Full Fine-tuning: An Illusion of Equivalence
  • [5] PEFT-Arena: Understanding Parameter-Efficient Finetuning from a Stability-Plasticity Perspective
  • [6] Qwen2.5 Technical Report

Dear Reviewer AAwM,

As the discussion period draws to a close, we would appreciate it if you could let us know whether our response addresses your concerns. We would be happy to clarify any remaining points. Thank you for taking the time to review our response.

Kind regards, The Authors

Reviewer_eC3VOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents an empirical comparison of six PEFT methods (LoRA, DoRA, PiSSA, MiLoRA, OFT and HRA) on Qwen2.5-7B, Llama-3.1-8B and FLUX-4B, covering math, coding and image personalization. It does not propose a new method. Instead, it asks whether differences in parameterization lead to real differences in task performance, retention of pretrained capabilities and weight geometry. Each method's learning rate is tuned separately, and results are averaged over three seeds. The paper reports three main findings. LoRA-family methods approximately preserve the pretrained singular-value spectrum and hyperspherical energy without explicit constraints, and restoring the pretrained singular values after training barely changes task performance. Task scores are often close, yet LoRA most consistently limits forgetting, DoRA reaches slightly higher mean task scores, and PiSSA often forgets more. Restoring the dominant singular components gives the largest reduction in forgetting for every method, while its effect on task performance varies across settings. My overall impression is that the experiments are careful, but most observations are already known or expected in this short fine-tuning regime, and the paper does not turn them into clear guidance for choosing or designing PEFT methods.

Critical strengths
  1. The question is of practical relevance. New PEFT methods are often compared under a fixed learning rate and a single seed. This paper tunes each method separately and uses three seeds, which gives a fairer comparison.
  2. The paper studies task performance, capability retention and weight geometry together. This shows that methods with similar task scores can still differ in forgetting, and that tighter geometric preservation does not always mean less forgetting.
  3. The spectral intervention experiments provide some interesting observations. In particular, restoring the dominant singular components consistently reduces forgetting across all six methods.
Critical weaknesses
  1. The novelty is unclear. Prior work has already discussed the core observations. Lee et al. (2026) and Schulman & Thinking Machines Lab (2025) show that tuned LoRA is competitive with its variants, Biderman et al. (2024) show that LoRA forgets less, and ETHER questions the need to preserve hyperspherical energy. Shuttleworth et al. (2025) and Jin et al. (2026) already link singular-vector changes to forgetting. The theory (Propositions 3.1 and 3.2) consists of classical results on the singular-value trace inequality and first-order perturbation.
  2. The new findings depend heavily on the experimental setting. OFT forgets least for Llama at the largest capacity, HRA forgets least at cat rank 8, and which spectral components suffice for the task changes with model, capacity and task for every method except PiSSA. The discussion gives no clear rule for which method to choose, so the paper reads more like a benchmark study.
  3. The setting is too narrow to support the geometric conclusions. Training uses 20,000 examples for one epoch, and the selected updates are small (about 0.15% to 0.3% of the weight norm for Qwen LoRA at medium and large capacity). Proposition 3.2 itself predicts small spectral drift for such updates. Restoring the singular values moves the weights by only about 1% to 11% of the update norm, so an unchanged task score is expected. Post-hoc restoration also does not show that geometric constraints are unnecessary during optimization.
  4. The forgetting comparison is not reliable enough. Learning rates are selected by task performance, but the selected configurations already show seed SDs of up to about 2 pp, so neighboring learning rates may be indistinguishable. Forgetting then depends mainly on which rate is chosen. For Qwen LoRA, the rank-7 selection uses 1e-3 and forgets the most in its setting (0.0237), while the rank-14 selection uses 1e-5 and forgets the least in its setting (0.0020). Claims such as DoRA beating LoRA in 10 of 14 comparisons count mean wins without significance tests (one-sided binomial p ≈ 0.09).
  5. Task gains and forgetting are measured narrowly. Forgetting in LLMs is measured only by NLL on 200 FineWiki documents. On Qwen the increases are 0.002 to 0.024 nats, about 0.2% to 2.4% in perplexity, and no downstream capability benchmark is reported. Base-model GSM8K accuracy is not reported, and Qwen gains less than 1 pp over the base model on HumanEval+ and MBPP+. Part of the "competitive performance" may therefore reflect little adaptation rather than equivalent methods.
Questions for authors
  1. Do the main conclusions still hold with longer fine-tuning and larger weight changes? In particular, does spectral drift stay small, does singular-value restoration still preserve task performance, and does LoRA still forget least? This is my main concern.
  2. How does full fine-tuning compare with the six methods on the same geometry and forgetting metrics? In addition, if the trained weights are perturbed by a random matrix with the same norm as the restoration edit, is task performance also unchanged?
  3. If the methods are compared at matched task performance, or with a selection rule robust to seed noise (e.g., the smallest learning rate within one SD of the best), does LoRA's retention advantage remain?
  4. Beyond the observations already reported in prior work, which finding do the authors consider the main original contribution, and what concrete recommendation does it give for choosing or designing PEFT methods?
4 comments

Thank you for the constructive review. We have conducted the related experiments and polished our conclusions. The major revision are highlighted in red in the updated version.

W1

1.1 What does the comparison add to prior work?

Prior SVD studies [2–4] and PEFT works [5–8] examined weight changes and tuning outcomes, but the studies were incomplete and unclear for PEFT, e.g., which certain spectrum matters most, and what is the quantitive trade-off between task performance and forgetting.

Instead, we provide a refined analysis on spectral properties of weight matrices in PEFT methods, and conduct a comprehensive study via AUC plots on performance-forgetting trade-off of different PEFT methods, see §§3–5 and Figures 3c–d and 4b for details.

We also summarized the relevant Fine-Tuning analysis papers and our work as follow for your reference.

Study and aimExperimental settingEvaluation metricsConclusions
He & Cao [2] analyze full finetuning weight change using SVDFully tuned LLMs, rescaled spectrum restorationAccuracy, singular-vector rotationRestoring the rescaled spectrum largely preserved accuracy. Leading directions rotated little, suggesting adaptation mainly involved other directions.
Jin et al. [3] — explain RL and SFT differenceFull-parameter SFT followed by RL, progressive restoration of leading directionsAccuracyRL recovered some accuracy under changed rules. Full-parameter SFT gains were concentrated mainly in leading directions.
Shuttleworth et al. [4] — explain LoRA forgetting compared with Full FTLoRA versus full tuning, attenuation of left singular directions with no close pretrained match (“intruders”)Accuracy, pretraining pseudo-lossAttenuating LoRA intruders reduced forgetting with little loss in task accuracy.
PEFT-Arena [1] — compare adaptation and retention across PEFT methodsCOT style Math and medical SFT, one shared LR and one seedTask scores, IFEval, NQ, BBH, weight and activation geometryOFT showed a favorable adaptation–retention trade-off, which the authors linked to preservation of weight and activation geometry.
Our work — identify what holds across PEFT methodsSix methods on language and diffusion tasks, per-method LR tuning over three seeds, spectral group restorationTask scores, FineWiki ΔNLL, image drift(1) There is no consistent conclusion for task performance. However, the top spectral components consistently matters the most on forgetting. (2) Tuned LoRA forgot least in 9/12 settings at competitive task scores. (3) OFT and HRA do not show consistently better accuracy-forgetting trade-off

1.2 The theory consists of classical results

We agree that the techniques behind our theory are standard, but the conclusion is non-intuitive. Our theoretical finding is that, under a small change of the weight matrix, changes on sigular values is a higher order of that on singular vectors. That means, singular vectors dominates the changes.

According to your suggestions, we have shorten the theory part to make it more informative and coincise.


W2 — Guidelines for practitioners

2.1 The new findings depend heavily on the experimental setting.

Based on your feedback, we conducted more experiments under the AUC metric for comprehensive comparisons.

We agree that LoRA not always lead to least forgetting at competitive accuracy. We revised the claim to be "LoRA most frequently limits forgetting at competitive task scores".

As for which spectral components suffice for the task, our finding is that there is no consistent conclusion for task performance. However, the top spectral components consistently matters the most on forgetting.

2.2 How should practitioners choose a method?

Our accuracy-forgetting AUC results can serve as guideline evidence for practitioners. DoRA has the best mean accuracy–forgetting AUC rank across the three setups, followed by tuned LoRA (Figures 3c,d and 4b).

When runtime matters, we recommend starting with tuned LoRA: it most frequently limited forgetting at competitive task scores and generally took less time per run than DoRA (Tables 4 and 14).

W3 + Q1

3.1 Can a small weight update improve accuracy?

Yes. For example, Qwen LoRA gained over 78 GSM8K test points with an update only 0.15% of the pretrained weight norm (Tables 2a and 21).

3.2 What happens to singular value drift, accuracy, and forgetting after longer training?

The conclusions on these patterns are mostly preserved from short training to longer training (e.g., 3x more steps), see Appendix B.7, Tables 19–20 and Figure 8.

3.3 What does a small restoration edit tell us?

As a diagnostic sanity check, restoring the singular values had little effect on downstream performance. This suggests that singular-value drift during fine-tuning contributed little to the final performance (§3, Figure 2 and Appendix C.2, Figure 9).

3.4 Post-hoc restoration alone does not show that geometric constraints are unnecessary during optimization.

Thank you for pointing this out. We updated our claim in §4, where OFT and HRA (with such geometric constraints) showed no consistent advantage over tuned LoRA in task performance or retention.


W4 + Q3

4.1 Does LoRA still forget less under the one-SD rule or at matched accuracy?

We sincerely thank the reviewer for suggesting the one-SD rule (i.e. the smallest learning rate within one SD of the best) which accounts for seed variability, to select method specific LR.

We have conducted the suggested experiment and found that LoRA forgot least in 7/8 language settings under the one-SD rule (Table 16) and forgot less in 35/43 Qwen and 31/35 Llama pairs matched on parameter budget and mean GSM8K dev accuracy (within 0.5 points, Appendix B.6).

4.2 What do DoRA’s 10/14 higher scores establish (p-value ≈ 0.09>0.05)?

We thank the reviewer for conducting the one-sided binomial test. The result (p=0.0898p=0.0898, Appendix B.6) is not significant at the 5% level, so we revised our DoRA claim.

Furthermore, we conducted the experiments under AUC metric, and find that tuned LoRA most frequently limit forgetting at competitive performance (Table 14), but DoRA achieves the best performance–forgetting trade-off under AUC, see (Figures 3c,d and 4b).


W5

5.1 How much did fine-tuning improve task scores?

Fine-tuning generally improved downstream performance. All LLM math training improved the base model over 60 pp accuracy and FLUX training improved 14 DINO points, with Qwen coding being the only exception (Tables 2 and 3a). Base Qwen already scored 13–20 points above base Llama, which may help explain its smaller gains (Table 2b).

5.2 Do small NLL differences agree with other retention measures?

The retention ranking held across 10,000 FineWiki documents, and LoRA had a higher mean PEFT-Arena general score [1] than OFT (Appendix B.1, Tables 9 and 6).


Q2

2.1 Comparison with full fine-tuning results on forgetting and geometry

Forgetting. Full fine-tuning forgot less than all adapters on Llama and all except LoRA on Qwen (Tables 7–8 and 27). On FLUX cat, full fine-tuning was uncompetitive on the fidelity–drift trade-off (Table 3a).

BackboneFull FT GSM8K dev (%)LoRA GSM8K dev (%)Full FT FineWiki-200 ΔNLLLoRA FineWiki-200 ΔNLL
Qwen92.20± 0.3592.20_{\pm\,0.35}91.40± 0.4491.40_{\pm\,0.44}0.00348± 0.000190.00348_{\pm\,0.00019}0.00201± 0.000140.00201_{\pm\,0.00014}
Llama72.87± 1.1672.87_{\pm\,1.16}72.30± 1.0472.30_{\pm\,1.04}0.01552± 0.000160.01552_{\pm\,0.00016}0.03723± 0.002300.03723_{\pm\,0.00230}

Geometry. On Qwen and Llama, singular-value drift under full fine-tuning was about 1% of the update norm, and restoring pretrained singular values does not change performance (Tables 27 and 32).

2.2 Do random edits of the same size preserve accuracy?

Norm-matched random edits also largely preserved accuracy. Gaussian edits matched to each projection’s singular-value reset norm changed mean GSM8K dev accuracy by at most 0.31 points across Llama methods (five draws per seed, three seeds, Appendix C.2, Table 28).

Q4 — What is the main original finding, and how should it guide PEFT selection or design?

Our main finding is: the top spectral components matters the most on forgetting, but there is no consistent conclusion for task performance.

This finding could inspire adapters that constrain updates to dominant directions, allocate rank across spectral groups, or learn which groups to update for each task. Their performance–forgetting trade-offs can be compared with tuned LoRA and DoRA using AUC across matched seeds.

Besides, we use AUC to evaluate the trade-off between task performance and forgetting. DoRA ranks first on average across three setups, followed by LoRA. At the LRs selected by task score, LoRA most frequently limits forgetting at competitive performance and generally runs faster than DoRA (Table 4).


References

  • [1] PEFT-Arena: Understanding Parameter-Efficient Finetuning from a Stability-Plasticity Perspective
  • [2] Understanding Post-Training Structural Changes in Large Language Models
  • [3] RL Fine-Tuning Heals OOD Forgetting in SFT
  • [4] LoRA vs Full Fine-tuning: An Illusion of Equivalence
  • [5] Learning Rate Matters: Vanilla LoRA May Suffice for LLM Fine-tuning
  • [6] LoRA Without Regret
  • [7] LoRA Learns Less and Forgets Less
  • [8] ETHER: Efficient Finetuning of Large-Scale Models with Hyperplane Reflections

Dear Reviewer ec3V,

As the discussion period draws to a close, we would appreciate it if you could let us know whether our response addresses your concerns. We would be happy to clarify any remaining points. Thank you for taking the time to review our response.

Kind regards, The Authors

Comments on the paper

2 comments
Area_Chair_i5ETOpenReview ↗

Discussion deadline approaching

Dear Reviewers and Authors,

Thank you for the detailed reviews and rebuttal.

To Reviewers ec3V and AAwM: As the author–reviewer discussion deadline approaches, could you please review the authors’ responses and indicate whether your concerns have been addressed?

Kind regards,

AC

We thank the three reviewers for their constructive feedback and the Area Chair for overseeing the review process. To support the assessment, we summarize the main revisions clarifying our contribution, strengthening comparisons, and addressing evaluation concerns.

1. Distinguishing our contribution. The closest prior spectral studies examine full fine-tuning or compare LoRA with full fine-tuning. Some prior explanations do not fully account for our observations, and some reported patterns conflict with our findings in the PEFT settings we test. We therefore examine which findings extend across PEFT methods in language and diffusion models. The spectral components supporting task performance vary by method and setting, while restoring dominant components consistently gives the largest mean reduction in forgetting.

2. Strengthening method comparisons. Our comparison uses method-specific learning-rate tuning and multiple seeds. We added AUC, alternative learning-rate selection and matched-accuracy comparisons. DoRA has the best average AUC rank across the three evaluated setups, followed by LoRA, while LoRA most frequently limits forgetting at competitive performance. OFT and HRA explicitly aim to preserve pretrained singular values, yet neither shows a consistently better performance–forgetting trade-off than tuned LoRA in the tested settings.

3. Testing robustness. We added evaluations with longer training, a larger dataset for measuring forgetting, and general-ability benchmarks. These additional checks broadly support our observations on task performance and forgetting.

4. Addressing scoring sensitivity. We document how strict scoring understated base-model accuracy and provide matched-prompt evaluations. We further addressed Reviewer AAwM’s follow-up questions by extending final-number scoring to the component interventions and full GSM8K development grid. Task gains and our main conclusions on spectral components and performance–forgetting trade-offs hold under final-number scoring.

We believe these substantive revisions strengthen the manuscript and make its contribution easier to assess. We appreciate the reviewers’ and Area Chair’s consideration of the revised manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.